mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 13:08:11 +00:00
Fix tables of contents, headers and formatting
This commit is contained in:
+6
-6
@@ -12,7 +12,7 @@ In this guide, we recommend best practices for cluster-level logging and applica
|
||||
- [Application Logging](#application-logging)
|
||||
- [General Best Practices](#general-best-practices)
|
||||
|
||||
# Changes in Logging in Rancher v2.5
|
||||
## Changes in Logging in Rancher v2.5
|
||||
|
||||
Before Rancher v2.5, logging in Rancher has historically been a pretty static integration. There were a fixed list of aggregators to choose from (ElasticSearch, Splunk, Kafka, Fluentd and Syslog), and only two configuration points to choose (Cluster-level and Project-level).
|
||||
|
||||
@@ -20,7 +20,7 @@ Logging in 2.5 has been completely overhauled to provide a more flexible experie
|
||||
|
||||
"Under the hood", Rancher logging uses the Banzai Cloud logging operator. We provide manageability of this operator (and its resources), and tie that experience in with managing your Rancher clusters.
|
||||
|
||||
# Cluster-level Logging
|
||||
## Cluster-level Logging
|
||||
|
||||
### Cluster-wide Scraping
|
||||
|
||||
@@ -38,7 +38,7 @@ Currently (as of v2.5.1) the logs from RKE containers are collected, but are not
|
||||
|
||||
A future release of Rancher will include the source container name which will enable filtering of these component logs. Once that change is made, you will be able to customize a _ClusterFlow_ to retrieve **only** the Kubernetes component logs, and direct them to an appropriate output.
|
||||
|
||||
# Application Logging
|
||||
## Application Logging
|
||||
|
||||
Best practice not only in Kubernetes but in all container-based applications is to direct application logs to `stdout`/`stderr`. The container runtime will then trap these logs and do **something** with them - typically writing them to a file. Depending on the container runtime (and its configuration), these logs can end up in any number of locations.
|
||||
|
||||
@@ -54,7 +54,7 @@ The goal of setting up a streaming sidecar is to take log files that are written
|
||||
|
||||
To set this up, edit your workload resource (e.g. Deployment) and add the following sidecar definition:
|
||||
|
||||
```
|
||||
```yaml
|
||||
...
|
||||
containers:
|
||||
- args:
|
||||
@@ -74,7 +74,7 @@ This will add a container to your workload definition that will now stream the c
|
||||
|
||||
This log stream is then automatically collected according to any _Flows_ or _ClusterFlows_ you have setup. You may also wish to consider creating a _Flow_ specifically for this log file by targeting the name of the container. See example:
|
||||
|
||||
```
|
||||
```yaml
|
||||
...
|
||||
spec:
|
||||
match:
|
||||
@@ -85,7 +85,7 @@ spec:
|
||||
```
|
||||
|
||||
|
||||
# General Best Practices
|
||||
## General Best Practices
|
||||
|
||||
- Where possible, output structured log entries (e.g. `syslog`, JSON). This makes handling of the log entry easier as there are already parsers written for these formats.
|
||||
- Try to provide the name of the application that is creating the log entry, in the entry itself. This can make troubleshooting easier as Kubernetes objects do not always carry the name of the application as the object name. For instance, a pod ID may be something like `myapp-098kjhsdf098sdf98` which does not provide much information about the application running inside the container.
|
||||
|
||||
+7
-14
@@ -10,19 +10,12 @@ Configuring sensible monitoring and alerting rules is vital for running any prod
|
||||
|
||||
The [Rancher monitoring documentation](../../../pages-for-subheaders/monitoring-and-alerting.md) describes how you can set up a complete Prometheus and Grafana stack. Out of the box this will scrape monitoring data from all system and Kubernetes components in your cluster and provide sensible dashboards and alerts for them to get started. But for a reliable setup, you also need to monitor your own workloads and adapt Prometheus and Grafana to your own specific use cases and cluster sizes. This document aims to give you best practices for this.
|
||||
|
||||
- [What to Monitor](#what-to-monitor)
|
||||
- [Configuring Prometheus Resource Usage](#configuring-prometheus-resource-usage)
|
||||
- [Scraping Custom Workloads](#scraping-custom-workloads)
|
||||
- [Monitoring in a (Micro)Service Architecture](#monitoring-in-a-micro-service-architecture)
|
||||
- [Real User Monitoring](#real-user-monitoring)
|
||||
- [Security Monitoring](#security-monitoring)
|
||||
- [Setting up Alerts](#setting-up-alerts)
|
||||
|
||||
# What to Monitor
|
||||
## What to Monitor
|
||||
|
||||
Kubernetes itself, as well as applications running inside of it, form a distributed system where different components interact with each other. For the whole system and each individual component, you have to ensure performance, availability, reliability and scalability. A good resource with more details and information is Google's free [Site Reliability Engineering Book](https://landing.google.com/sre/sre-book/), especially the chapter about [Monitoring distributed systems](https://landing.google.com/sre/sre-book/chapters/monitoring-distributed-systems/).
|
||||
|
||||
# Configuring Prometheus Resource Usage
|
||||
## Configuring Prometheus Resource Usage
|
||||
|
||||
When installing the integrated monitoring stack, Rancher allows to configure several settings that are dependent on the size of your cluster and the workloads running in it. This chapter covers these in more detail.
|
||||
|
||||
@@ -66,7 +59,7 @@ Prometheus is not meant to store metrics for a long amount of time, but should o
|
||||
|
||||
In order to store some, or all metrics for a long time, you can leverage Prometheus' [remote read/write](https://prometheus.io/docs/prometheus/latest/storage/#remote-storage-integrations) capabilities to connect it to storage systems like [Thanos](https://thanos.io/), [InfluxDB](https://www.influxdata.com/), [M3DB](https://www.m3db.io/), or others. You can find an example setup in this [blog post](https://rancher.com/blog/2020/prometheus-metric-federation).
|
||||
|
||||
# Scraping Custom Workloads
|
||||
## Scraping Custom Workloads
|
||||
|
||||
While the integrated Rancher Monitoring already scrapes system metrics from a cluster's nodes and system components, the custom workloads that you deploy on Kubernetes should also be scraped for data. For that you can configure Prometheus to do an HTTP request to an endpoint of your applications in a certain interval. These endpoints should then return their metrics in a Prometheus format.
|
||||
|
||||
@@ -94,23 +87,23 @@ To still get metrics for these use cases, you can set up [prometheus-pushgateway
|
||||
|
||||
Sometimes it is useful to monitor workloads from the outside. For this, you can use the [Prometheus blackbox-exporter](https://github.com/prometheus/blackbox_exporter) which allows probing any kind of endpoint over HTTP, HTTPS, DNS, TCP and ICMP.
|
||||
|
||||
# Monitoring in a (Micro)Service Architecture
|
||||
## Monitoring in a (Micro)Service Architecture
|
||||
|
||||
If you have a (micro)service architecture where multiple individual workloads within your cluster are communicating with each other, it is really important to have detailed metrics and traces about this traffic to understand how all these workloads are communicating with each other and where a problem or bottleneck may be.
|
||||
|
||||
Of course you can monitor all this internal traffic in all your workloads and expose these metrics to Prometheus. But this can quickly become quite work intensive. Service Meshes like Istio, which can be installed with [a click](../../../pages-for-subheaders/istio.md) in Rancher, can do this automatically and provide rich telemetry about the traffic between all services.
|
||||
|
||||
# Real User Monitoring
|
||||
## Real User Monitoring
|
||||
|
||||
Monitoring the availability and performance of all your internal workloads is vitally important to run stable, reliable and fast applications. But these metrics only show you parts of the picture. To get a complete view it is also necessary to know how your end users are actually perceiving it. For this you can look into various [Real user monitoring solutions](https://en.wikipedia.org/wiki/Real_user_monitoring).
|
||||
|
||||
# Security Monitoring
|
||||
## Security Monitoring
|
||||
|
||||
In addition to monitoring workloads to detect performance, availability or scalability problems, the cluster and the workloads running into it should also be monitored for potential security problems. A good starting point is to frequently run and alert on [CIS Scans](../../../pages-for-subheaders/cis-scans.md) which check if the cluster is configured according to security best practices.
|
||||
|
||||
For the workloads, you can have a look at Kubernetes and Container security solutions like [Falco](https://falco.org/), [Aqua Kubernetes Security](https://www.aquasec.com/solutions/kubernetes-container-security/), [SysDig](https://sysdig.com/).
|
||||
|
||||
# Setting up Alerts
|
||||
## Setting up Alerts
|
||||
|
||||
Getting all the metrics into a monitoring systems and visualizing them in dashboards is great, but you also want to be pro-actively alerted if something goes wrong.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user