mirror of
https://github.com/rancher/rancher-docs.git
synced 2026-09-26 04:58:07 +00:00
Remove unneeded intermediate folders
This commit is contained in:
Billy Tat
@@ -0,0 +1,252 @@
|
||||
---
|
||||
title: Project Alerts
|
||||
weight: 2526
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/legacy/alerts/project-alerts
|
||||
- /rancher/v2.0-v2.4/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/project-alerts
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-alerts/project-alerts/
|
||||
---
|
||||
|
||||
To keep your clusters and applications healthy and driving your organizational productivity forward, you need to stay informed of events occurring in your clusters and projects, both planned and unplanned. When an event occurs, your alert is triggered, and you are sent a notification. You can then, if necessary, follow up with corrective actions.
|
||||
|
||||
Notifiers and alerts are built on top of the [Prometheus Alertmanager](https://prometheus.io/docs/alerting/alertmanager/). Leveraging these tools, Rancher can notify [cluster owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles) and [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) of events they need to address.
|
||||
|
||||
Before you can receive alerts, one or more [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) must be configured at the cluster level.
|
||||
|
||||
Only [administrators]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/), [cluster owners or members]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), or [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can manage project alerts.
|
||||
|
||||
This section covers the following topics:
|
||||
|
||||
- [Alerts scope](#alerts-scope)
|
||||
- [Default project-level alerts](#default-project-level-alerts)
|
||||
- [Adding project alerts](#adding-project-alerts)
|
||||
- [Managing project alerts](#managing-project-alerts)
|
||||
- [Project Alert Rule Configuration](#project-alert-rule-configuration)
|
||||
- [Pod Alerts](#pod-alerts)
|
||||
- [Workload Alerts](#workload-alerts)
|
||||
- [Workload Selector Alerts](#workload-selector-alerts)
|
||||
- [Metric Expression Alerts](#metric-expression-alerts)
|
||||
|
||||
|
||||
# Alerts Scope
|
||||
|
||||
The scope for alerts can be set at either the [cluster level]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/) or project level.
|
||||
|
||||
At the project level, Rancher monitors specific deployments and sends alerts for:
|
||||
|
||||
* Deployment availability
|
||||
* Workloads status
|
||||
* Pod status
|
||||
* The Prometheus expression cross the thresholds
|
||||
|
||||
# Default Project-level Alerts
|
||||
|
||||
When you enable monitoring for the project, some project-level alerts are provided. You can receive these alerts if a [notifier]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) for them is configured at the cluster level.
|
||||
|
||||
| Alert | Explanation |
|
||||
|-------|-------------|
|
||||
| Less than half workload available | A critical alert is triggered if less than half of a workload is available, based on workloads where the key is `app` and the value is `workload`. |
|
||||
| Memory usage close to the quota | A warning alert is triggered if the workload's memory usage exceeds the memory resource quota that is set for the workload. You can see the memory limit in the Rancher UI if you go to the workload under the **Security & Host Config** tab. |
|
||||
|
||||
For information on other default alerts, refer to the section on [cluster-level alerts.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts/default-alerts)
|
||||
|
||||
# Adding Project Alerts
|
||||
|
||||
>**Prerequisite:** Before you can receive project alerts, you must add a notifier.
|
||||
|
||||
1. From the **Global** view, navigate to the project that you want to configure project alerts for. Select **Tools > Alerts**. In versions before v2.2.0, you can choose **Resources > Alerts**.
|
||||
|
||||
1. Click **Add Alert Group**.
|
||||
|
||||
1. Enter a **Name** for the alert that describes its purpose, you could group alert rules for the different purpose.
|
||||
|
||||
1. Based on the type of alert you want to create, fill out the form. For help, refer to the [configuration](#project-alert-rule-configuration) section below.
|
||||
|
||||
1. Continue adding more alert rules to the group.
|
||||
|
||||
1. Finally, choose the [notifiers]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers/) that send you alerts.
|
||||
|
||||
- You can set up multiple notifiers.
|
||||
- You can change notifier recipients on the fly.
|
||||
|
||||
1. Click **Create.**
|
||||
|
||||
**Result:** Your alert is configured. A notification is sent when the alert is triggered.
|
||||
|
||||
|
||||
# Managing Project Alerts
|
||||
|
||||
To manage project alerts, browse to the project that alerts you want to manage. Then select **Tools > Alerts**. In versions before v2.2.0, you can choose **Resources > Alerts**. You can:
|
||||
|
||||
- Deactivate/Reactive alerts
|
||||
- Edit alert settings
|
||||
- Delete unnecessary alerts
|
||||
- Mute firing alerts
|
||||
- Unmute muted alerts
|
||||
|
||||
|
||||
# Project Alert Rule Configuration
|
||||
|
||||
- [Pod Alerts](#pod-alerts)
|
||||
- [Workload Alerts](#workload-alerts)
|
||||
- [Workload Selector Alerts](#workload-selector-alerts)
|
||||
- [Metric Expression Alerts](#metric-expression-alerts)
|
||||
|
||||
# Pod Alerts
|
||||
|
||||
This alert type monitors for the status of a specific pod.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select the **Pod** option, and then select a pod from the drop-down.
|
||||
|
||||
### Is
|
||||
|
||||
Select a pod status that triggers an alert:
|
||||
|
||||
- **Not Running**
|
||||
- **Not Scheduled**
|
||||
- **Restarted <x> times within the last <x> Minutes**
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert. The options are:
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on pod state. For example, select **Info** for Job pod which stop running after job finished. However, if an important pod isn't scheduled, it may affect operations, so choose **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group.
|
||||
|
||||
You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
|
||||
# Workload Alerts
|
||||
|
||||
This alert type monitors for the availability of a workload.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Choose the **Workload** option. Then choose a workload from the drop-down.
|
||||
|
||||
### Is
|
||||
|
||||
Choose an availability percentage using the slider. The alert is triggered when the workload's availability on your cluster nodes drops below the set percentage.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on the percentage you choose and the importance of the workload.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group.
|
||||
|
||||
You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
|
||||
# Workload Selector Alerts
|
||||
|
||||
This alert type monitors for the availability of all workloads marked with tags that you've specified.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When a
|
||||
|
||||
Select the **Workload Selector** option, and then click **Add Selector** to enter the key value pair for a label. If one of the workloads drops below your specifications, an alert is triggered. This label should be applied to one or more of your workloads.
|
||||
|
||||
### Is
|
||||
|
||||
Choose an availability percentage using the slider. The alert is triggered when the workload's availability on your cluster nodes drops below the set percentage.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on the percentage you choose and the importance of the workload.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group.
|
||||
|
||||
You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
|
||||
# Metric Expression Alerts
|
||||
_Available as of v2.2.4_
|
||||
|
||||
If you enable [project monitoring]({{<baseurl>}}/rancher/v2.0-v2.4/en/project-admin/tools/#monitoring), this alert type monitors for the overload from Prometheus expression querying.
|
||||
|
||||
Each of the below sections corresponds to a part of the alert rule configuration section in the Rancher UI.
|
||||
|
||||
### When A
|
||||
|
||||
Input or select an **Expression**. The dropdown shows the original metrics from Prometheus, including:
|
||||
|
||||
- [**Container**](https://github.com/google/cadvisor)
|
||||
- [**Kubernetes Resources**](https://github.com/kubernetes/kube-state-metrics)
|
||||
- **Customize**
|
||||
- [**Project Level Grafana**](http://docs.grafana.org/administration/metrics/)
|
||||
- **Project Level Prometheus**
|
||||
|
||||
### Is
|
||||
|
||||
Choose a comparison.
|
||||
|
||||
- **Equal**: Trigger alert when expression value equal to the threshold.
|
||||
- **Not Equal**: Trigger alert when expression value not equal to the threshold.
|
||||
- **Greater Than**: Trigger alert when expression value greater than to threshold.
|
||||
- **Less Than**: Trigger alert when expression value equal or less than the threshold.
|
||||
- **Greater or Equal**: Trigger alert when expression value greater to equal to the threshold.
|
||||
- **Less or Equal**: Trigger alert when expression value less or equal to the threshold.
|
||||
|
||||
If applicable, choose a comparison value or a threshold for the alert to be triggered.
|
||||
|
||||
### For
|
||||
|
||||
Select a duration for a trigger alert when the expression value crosses the threshold longer than the configured duration.
|
||||
|
||||
### Send a
|
||||
|
||||
Select the urgency level of the alert.
|
||||
|
||||
- **Critical**: Most urgent
|
||||
- **Warning**: Normal urgency
|
||||
- **Info**: Least urgent
|
||||
|
||||
Select the urgency level of the alert based on its impact on operations. For example, an alert triggered when a expression for container memory close to the limit raises above 60% deems an urgency of **Info**, but raised about 95% deems an urgency of **Critical**.
|
||||
|
||||
### Advanced Options
|
||||
|
||||
By default, the below options will apply to all alert rules within the group. You can disable these advanced options when configuring a specific rule.
|
||||
|
||||
- **Group Wait Time**: How long to wait to buffer alerts of the same group before sending initially, default to 30 seconds.
|
||||
- **Group Interval Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 30 seconds.
|
||||
- **Repeat Wait Time**: How long to wait before sending an alert that has been added to a group which contains already fired alerts, default to 1 hour.
|
||||
+115
@@ -0,0 +1,115 @@
|
||||
---
|
||||
title: Project Logging
|
||||
shortTitle: Project Logging
|
||||
weight: 2527
|
||||
aliases:
|
||||
- /rancher/v2.0-v2.4/en/project-admin/tools/logging
|
||||
- /rancher/v2.0-v2.4/en/logging/legacy/project-logging
|
||||
- /rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/project-logging
|
||||
- /rancher/v2.x/en/logging/v2.0.x-v2.4.x/project-logging/
|
||||
- /rancher/v2.x/en/monitoring-alerting/v2.0.x-v2.4.x/cluster-monitoring/project-monitoring/
|
||||
---
|
||||
|
||||
Rancher can integrate with a variety of popular logging services and tools that exist outside of your Kubernetes clusters.
|
||||
|
||||
For background information about how logging integrations work, refer to the [cluster administration section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/logging/v2.0.x-v2.4.x/cluster-logging/#how-logging-integrations-work)
|
||||
|
||||
Rancher supports the following services:
|
||||
|
||||
- Elasticsearch
|
||||
- Splunk
|
||||
- Kafka
|
||||
- Syslog
|
||||
- Fluentd
|
||||
|
||||
>**Note:** You can only configure one logging service per cluster or per project.
|
||||
|
||||
Only [administrators]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/global-permissions/), [cluster owners or members]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#cluster-roles), or [project owners]({{<baseurl>}}/rancher/v2.0-v2.4/en/admin-settings/rbac/cluster-project-roles/#project-roles) can configure Rancher to send Kubernetes logs to a logging service.
|
||||
|
||||
# Requirements
|
||||
|
||||
The Docker daemon on each node in the cluster should be [configured](https://docs.docker.com/config/containers/logging/configure/) with the (default) log-driver: `json-file`. You can check the log-driver by running the following command:
|
||||
|
||||
```
|
||||
$ docker info | grep 'Logging Driver'
|
||||
Logging Driver: json-file
|
||||
```
|
||||
|
||||
# Advantages
|
||||
|
||||
Setting up a logging service to collect logs from your cluster/project has several advantages:
|
||||
|
||||
- Logs errors and warnings in your Kubernetes infrastructure to a stream. The stream informs you of events like a container crashing, a pod eviction, or a node dying.
|
||||
- Allows you to capture and analyze the state of your cluster and look for trends in your environment using the log stream.
|
||||
- Helps you when troubleshooting or debugging.
|
||||
- Saves your logs to a safe location outside of your cluster, so that you can still access them even if your cluster encounters issues.
|
||||
|
||||
# Logging Scope
|
||||
|
||||
You can configure logging at either cluster level or project level.
|
||||
|
||||
- [Cluster logging]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/) writes logs for every pod in the cluster, i.e. in all the projects. For [RKE clusters]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-provisioning/rke-clusters), it also writes logs for all the Kubernetes system components.
|
||||
|
||||
- Project logging writes logs for every pod in that particular project.
|
||||
|
||||
Logs that are sent to your logging service are from the following locations:
|
||||
|
||||
- Pod logs stored at `/var/log/containers`.
|
||||
|
||||
- Kubernetes system components logs stored at `/var/lib/rancher/rke/logs/`.
|
||||
|
||||
# Enabling Project Logging
|
||||
|
||||
1. From the **Global** view, navigate to the project that you want to configure project logging.
|
||||
|
||||
1. Select **Tools > Logging** in the navigation bar. In versions before v2.2.0, you can choose **Resources > Logging**.
|
||||
|
||||
1. Select a logging service and enter the configuration. Refer to the specific service for detailed configuration. Rancher supports the following services:
|
||||
|
||||
- [Elasticsearch]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/elasticsearch/)
|
||||
- [Splunk]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/splunk/)
|
||||
- [Kafka]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/kafka/)
|
||||
- [Syslog]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/syslog/)
|
||||
- [Fluentd]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging/fluentd/)
|
||||
|
||||
1. (Optional) Instead of using the UI to configure the logging services, you can enter custom advanced configurations by clicking on **Edit as File**, which is located above the logging targets. This link is only visible after you select a logging service.
|
||||
|
||||
- With the file editor, enter raw fluentd configuration for any logging service. Refer to the documentation for each logging service on how to setup the output configuration.
|
||||
|
||||
- [Elasticsearch Documentation](https://github.com/uken/fluent-plugin-elasticsearch)
|
||||
- [Splunk Documentation](https://github.com/fluent/fluent-plugin-splunk)
|
||||
- [Kafka Documentation](https://github.com/fluent/fluent-plugin-kafka)
|
||||
- [Syslog Documentation](https://github.com/dlackty/fluent-plugin-remote_syslog)
|
||||
- [Fluentd Documentation](https://docs.fluentd.org/v1.0/articles/out_forward)
|
||||
|
||||
- If the logging service is using TLS, you also need to complete the **SSL Configuration** form.
|
||||
1. Provide the **Client Private Key** and **Client Certificate**. You can either copy and paste them or upload them by using the **Read from a file** button.
|
||||
|
||||
- You can use either a self-signed certificate or one provided by a certificate authority.
|
||||
|
||||
- You can generate a self-signed certificate using an openssl command. For example:
|
||||
|
||||
```
|
||||
openssl req -x509 -newkey rsa:2048 -keyout myservice.key -out myservice.cert -days 365 -nodes -subj "/CN=myservice.example.com"
|
||||
```
|
||||
2. If you are using a self-signed certificate, provide the **CA Certificate PEM**.
|
||||
|
||||
1. (Optional) Complete the **Additional Logging Configuration** form.
|
||||
|
||||
1. **Optional:** Use the **Add Field** button to add custom log fields to your logging configuration. These fields are key value pairs (such as `foo=bar`) that you can use to filter the logs from another system.
|
||||
|
||||
1. Enter a **Flush Interval**. This value determines how often [Fluentd](https://www.fluentd.org/) flushes data to the logging server. Intervals are measured in seconds.
|
||||
|
||||
1. **Include System Log**. The logs from pods in system project and RKE components will be sent to the target. Uncheck it to exclude the system logs.
|
||||
|
||||
1. Click **Test**. Rancher sends a test log to the service.
|
||||
|
||||
> **Note:** This button is replaced with _Dry Run_ if you are using the custom configuration editor. In this case, Rancher calls the fluentd dry run command to validate the configuration.
|
||||
|
||||
1. Click **Save**.
|
||||
|
||||
**Result:** Rancher is now configured to send logs to the selected service. Log into the logging service so that you can start viewing the logs.
|
||||
|
||||
# Related Links
|
||||
|
||||
[Logging Architecture](https://kubernetes.io/docs/concepts/cluster-administration/logging/)
|
||||
@@ -0,0 +1,46 @@
|
||||
---
|
||||
title: Tools for Logging, Monitoring, and More
|
||||
weight: 2525
|
||||
---
|
||||
|
||||
Rancher contains a variety of tools that aren't included in Kubernetes to assist in your DevOps operations. Rancher can integrate with external services to help your clusters run more efficiently. Tools are divided into following categories:
|
||||
<!-- TOC -->
|
||||
|
||||
- [Notifiers](#notifiers)
|
||||
- [Alerts](#alerts)
|
||||
- [Logging](#logging)
|
||||
- [Monitoring](#monitoring)
|
||||
|
||||
<!-- /TOC -->
|
||||
|
||||
# Notifiers
|
||||
|
||||
[Notifiers]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/notifiers) are services that inform you of alert events. You can configure notifiers to send alert notifications to staff best suited to take corrective action. Notifications can be sent with Slack, email, PagerDuty, WeChat, and webhooks.
|
||||
|
||||
# Alerts
|
||||
|
||||
[Alerts]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/alerts) are rules that trigger notifications. Before you can receive alerts, you must configure one or more notifier in Rancher. The scope for alerts can be set at either the cluster or project level.
|
||||
|
||||
For details on project-level alerts, see [this page.](./project-alerts)
|
||||
|
||||
# Logging
|
||||
|
||||
Logging is helpful because it allows you to:
|
||||
|
||||
- Capture and analyze the state of your cluster
|
||||
- Look for trends in your environment
|
||||
- Save your logs to a safe location outside of your cluster
|
||||
- Stay informed of events like a container crashing, a pod eviction, or a node dying
|
||||
- More easily debug and troubleshoot problems
|
||||
|
||||
Rancher can integrate with Elasticsearch, splunk, kafka, syslog, and fluentd.
|
||||
|
||||
For details on setting up logging at the cluster level, refer to the [logging section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/logging)
|
||||
|
||||
For details on project-level logging, see [this section.](./project-logging)
|
||||
|
||||
# Monitoring
|
||||
|
||||
_Available as of v2.2.0_
|
||||
|
||||
Using Rancher, you can monitor the state and processes of your cluster nodes, Kubernetes components, and software deployments through integration with [Prometheus](https://prometheus.io/), a leading open-source monitoring solution. For details, refer to the [monitoring section.]({{<baseurl>}}/rancher/v2.0-v2.4/en/cluster-admin/tools/cluster-monitoring)
|
||||
Reference in New Issue
Block a user