Merge branch 'main' into ihm/251217-fix-docs-shared-shortcode
This commit is contained in:
Isabel Matwawana
@@ -428,12 +428,25 @@ Or using a Kubernetes format, for example `kubernetes-dashboard.json`:
|
||||
|
||||
You _must_ use the Kubernetes resource format to provision dashboards v2 / dynamic dashboards.
|
||||
|
||||
It later polls that path every `updateIntervalSeconds` for updates to the dashboard files and updates its database.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Grafana installs dashboards at the root level if you don't set the `folder` field.
|
||||
{{< /admonition >}}
|
||||
|
||||
#### Detect updates to provisioned dashboards files
|
||||
|
||||
After Grafana provisions your dashboards, it checks the filesystem for changes and updates dashboards as needed.
|
||||
|
||||
The mechanism Grafana uses to do this depends on your `updateIntervalSeconds` value:
|
||||
|
||||
- **More than 10 seconds**: Grafana polls the path at that interval.
|
||||
- **10 seconds or less**: Grafana watches the filesystem for changes and updates dashboards when it detects them.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
When `updateIntervalSeconds` is 10 or less, Grafana relies on filesystem watch events to detect changes.
|
||||
Depending on your filesystem and how you mount or sync dashboard files (for example, Docker bind mounts or some network filesystems), those events might not reach Grafana.
|
||||
To work around this, set `updateIntervalSeconds` to more than 10 to force polling, or update your setup so filesystem watch events are propagated.
|
||||
{{< /admonition >}}
|
||||
|
||||
#### Make changes to a provisioned dashboard
|
||||
|
||||
You can make changes to a provisioned dashboard in the Grafana UI but its not possible to automatically save the changes back to the provisioning source.
|
||||
|
||||
@@ -134,7 +134,7 @@ To convert data source-managed alert rules to Grafana managed alerts:
|
||||
|
||||
Pausing stops alert rule evaluation behavior for the newly created Grafana-managed alert rules.
|
||||
|
||||
9. (Optional) In the **Target data source** of the **Recording rules** section, you can select the data source that the imported recording rules will query. By default, it is the data source selected in the **Data source** dropdown.
|
||||
9. (Optional) In the **Target data source** of the **Recording rules** section, you can select the data source to which the imported recording rules will write metrics. By default, it is the data source selected in the **Data source** dropdown.
|
||||
|
||||
10. Click **Import**.
|
||||
|
||||
@@ -242,6 +242,8 @@ Set to `true` to import recording rules in paused state.
|
||||
|
||||
The UID of the data source to use for alert rule queries.
|
||||
|
||||
If not specified in the header, Grafana uses the configured default from `unified_alerting.prometheus_conversion.default_datasource_uid`. If neither the header nor the configuration option is provided, the request fails.
|
||||
|
||||
#### `X-Grafana-Alerting-Target-Datasource-UID`
|
||||
|
||||
The UID of the target data source for recording rules. If not specified, the value from `X-Grafana-Alerting-Datasource-UID` is used.
|
||||
|
||||
@@ -48,6 +48,14 @@ Recording rules can be helpful in various scenarios, such as:
|
||||
|
||||
The evaluation group of the recording rule determines how often the metric is pre-computed.
|
||||
|
||||
## Recommendations
|
||||
|
||||
- **Use frequent evaluation intervals**. Set frequent evaluation intervals for recording rules. Long intervals, such as an hour, can cause the recorded metric to be stale and lead to misaligned alert rule evaluations, especially when combined with a long pending period.
|
||||
- **Align alert evaluation with recording frequency**. The evaluation interval of an alert rule that depends on a recorded metric should be aligned with the recording rule's interval. If a recording rule runs every 3 minutes, the alert rule should also be evaluated at a similar frequency to ensure it acts on fresh data.
|
||||
- **Use `_over_time` functions for instant queries**. Since all alert rules are ultimately executed as an instant query, you can use functions like `max_over_time(my_metric[5m])` as an instant query. This allows you to get an aggregated value over a period without using a range query and a reduce expression.
|
||||
|
||||
## Types of recording rules
|
||||
|
||||
Similar to alert rules, Grafana supports two types of recording rules:
|
||||
|
||||
1. [Grafana-managed recording rules](ref:grafana-managed-recording-rules), which can query any Grafana data source supported by alerting. It's the recommended option.
|
||||
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/
|
||||
description: This section provides a set of guides for useful alerting practices and recommendations
|
||||
keywords:
|
||||
- grafana
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Best practices
|
||||
title: Grafana Alerting best practices
|
||||
weight: 170
|
||||
---
|
||||
|
||||
# Grafana Alerting best practices
|
||||
|
||||
This section provides a set of guides and examples of best practices for Grafana Alerting. Here you can learn more about how to handle common alert management problems and you can see examples of more advanced usage of Grafana Alerting.
|
||||
|
||||
{{< section >}}
|
||||
|
||||
Designing and configuring an alert management set up that works takes time. Here are some additional tips on how to create an effective alert management set up:
|
||||
|
||||
{{< shared id="alert-planning-fundamentals" >}}
|
||||
|
||||
**Which are the key metrics for your business that you want to monitor and alert on?**
|
||||
|
||||
- Find events that are important to know about and not so trivial or frequent that recipients ignore them.
|
||||
- Alerts should only be created for big events that require immediate attention or intervention.
|
||||
- Consider quality over quantity.
|
||||
|
||||
**How do you want to organize your alerts and notifications?**
|
||||
|
||||
- Be selective about who you set to receive alerts. Consider sending them to the right teams, whoever is on call, and the specific channels.
|
||||
- Think carefully about priority and severity levels.
|
||||
- Automate as far as possible provisioning Alerting resources with the API or Terraform.
|
||||
|
||||
**Which information should you include in notifications?**
|
||||
|
||||
- Consider who the alert receivers and responders are.
|
||||
- Share information that helps responders identify and address potential issues.
|
||||
- Link alerts to dashboards to guide responders on which data to investigate.
|
||||
|
||||
**How can you reduce alert fatigue?**
|
||||
|
||||
- Avoid noisy, unnecessary alerts by using silences, mute timings, or pausing alert rule evaluation.
|
||||
- Continually tune your alert rules to review effectiveness. Remove alert rules to avoid duplication or ineffective alerts.
|
||||
- Continually review your thresholds and evaluation rules.
|
||||
|
||||
**How should you configure recording rules?**
|
||||
|
||||
- Use frequent evaluation intervals. It is recommended to set a frequent evaluation interval for recording rules. Long intervals, such as an hour, can cause the recorded metric to be stale and lead to misaligned alert rule evaluations, especially when combined with a long pending period.
|
||||
- Understand query types. Grafana Alerting uses both **Instant** and **Range** queries. Instant queries fetch a single data point, while Range queries fetch a series of data points over time. When using a Range query in an alert condition, you must use a Reduce expression to aggregate the series into a single value.
|
||||
- Align alert evaluation with recording frequency. The evaluation interval of an alert rule that depends on a recorded metric should be aligned with the recording rule's interval. If a recording rule runs every 3 minutes, the alert rule should also be evaluated at a similar frequency to ensure it acts on fresh data.
|
||||
- Use `_over_time` functions for instant queries. Since all alert rules are ultimately executed as an instant query, you can use functions like `max_over_time(my_metric[1h])` as an instant query. This allows you to get an aggregated value over a period without using a range query and a reduce expression.
|
||||
|
||||
{{< /shared >}}
|
||||
@@ -0,0 +1,22 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/
|
||||
description: This section provides a set of guides for useful alerting practices and recommendations
|
||||
keywords:
|
||||
- grafana
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples
|
||||
title: Examples
|
||||
weight: 180
|
||||
---
|
||||
|
||||
# Examples
|
||||
|
||||
This section provides practical examples that show how to work with different types of alerting data, apply alert design patterns, reuse alert logic, and take advantage of specific Grafana Alerting features.
|
||||
|
||||
This section includes:
|
||||
|
||||
{{< section >}}
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/dynamic-labels
|
||||
aliases:
|
||||
- ../best-practices/dynamic-labels/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/dynamic-labels/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/dynamic-labels
|
||||
description: This example shows how to define dynamic labels based on query values, along with important behavior to keep in mind when using them.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -10,7 +12,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples of dynamic labels
|
||||
menuTitle: Dynamic labels
|
||||
title: Example of dynamic labels in alert instances
|
||||
weight: 1104
|
||||
refs:
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/dynamic-thresholds
|
||||
aliases:
|
||||
- ../best-practices/dynamic-thresholds/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/dynamic-thresholds/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/dynamic-thresholds
|
||||
description: This example shows how to use a distinct threshold value per dimension using multi-dimensional alerts and a Math expression.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -10,7 +12,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples of dynamic thresholds
|
||||
menuTitle: Dynamic thresholds
|
||||
title: Example of dynamic thresholds per dimension
|
||||
weight: 1105
|
||||
refs:
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/high-cardinality-alerts/
|
||||
aliases:
|
||||
- ../best-practices/high-cardinality-alerts/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/high-cardinality-alerts/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/high-cardinality-alerts/
|
||||
description: Learn how to detect and alert on high-cardinality metrics that can overload your metrics backend and increase observability costs.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -8,7 +10,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples of high-cardinality alerts
|
||||
menuTitle: High-cardinality alerts
|
||||
title: Examples of high-cardinality alerts
|
||||
weight: 1105
|
||||
refs:
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/multi-dimensional-alerts/
|
||||
aliases:
|
||||
- ../best-practices/multi-dimensional-alerts/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/multi-dimensional-alerts/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/multi-dimensional-alerts/
|
||||
description: This example shows how a single alert rule can generate multiple alert instances using time series data.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -8,7 +10,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples of multi-dimensional alerts
|
||||
menuTitle: Multi-dimensional alerts
|
||||
title: Example of multi-dimensional alerts on time series data
|
||||
weight: 1101
|
||||
refs:
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/table-data
|
||||
aliases:
|
||||
- ../best-practices/table-data/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/table-data/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/table-data
|
||||
description: This example shows how to create an alert rule using table data.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -8,7 +10,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Examples of table data
|
||||
menuTitle: Table data
|
||||
title: Example of alerting on tabular data
|
||||
weight: 1102
|
||||
refs:
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/trace-based-alerts/
|
||||
aliases:
|
||||
- ../best-practices/trace-based-alerts/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/trace-based-alerts/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/trace-based-alerts/
|
||||
description: This guide provides introductory examples and distinct approaches for setting up trace-based alerts in Grafana.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -8,7 +10,7 @@ labels:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
title: Examples of trace-based alerts
|
||||
title: Trace-based alerts
|
||||
weight: 1103
|
||||
refs:
|
||||
testdata-data-source:
|
||||
+3
-1
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/tutorials/
|
||||
aliases:
|
||||
- ../best-practices/tutorials/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/tutorials/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/examples/tutorials/
|
||||
description: This section provides a set of step-by-step tutorials guides to get started with Grafana Aletings.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -0,0 +1,35 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/guides/
|
||||
description: This section provides a set of guides for useful alerting practices and recommendations
|
||||
keywords:
|
||||
- grafana
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Guides
|
||||
title: Guides
|
||||
weight: 170
|
||||
refs:
|
||||
examples:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/examples/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/examples/
|
||||
tutorials:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/examples/tutorials/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/examples/tutorials/
|
||||
---
|
||||
|
||||
# Guides
|
||||
|
||||
Guides in the Grafana Alerting documentation provide best practices and practical recommendations to help you move from a basic alerting setup to real-world use cases.
|
||||
|
||||
These guides cover topics such as:
|
||||
|
||||
{{< section >}}
|
||||
|
||||
For more hands-on examples, refer to [Examples](ref:examples) and [Tutorials](ref:tutorials).
|
||||
@@ -0,0 +1,201 @@
|
||||
---
|
||||
aliases:
|
||||
- ../best-practices/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/guides/best-practices/
|
||||
description: Designing and configuring an effective alerting system takes time. This guide focuses on building alerting systems that scale with real-world operations.
|
||||
keywords:
|
||||
- grafana
|
||||
- alerting
|
||||
- guide
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Best practices
|
||||
title: Best practices
|
||||
weight: 1010
|
||||
refs:
|
||||
recovery-threshold:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/queries-conditions/#recovery-threshold
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/fundamentals/alert-rules/queries-conditions/#recovery-threshold
|
||||
keep-firing-for:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#keep-firing-for
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/fundamentals/alert-rule-evaluation/#keep-firing-for
|
||||
pending-period:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rule-evaluation/#pending-period
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/fundamentals/alert-rule-evaluation/#pending-period
|
||||
silences:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/configure-notifications/create-silence/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/configure-notifications/create-silence/
|
||||
timing-options:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/#timing-options
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/#timing-options
|
||||
group-alert-notifications:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/group-alert-notifications/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/group-alert-notifications/
|
||||
notification-policies:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/notifications/notification-policies/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/notifications/notification-policies/
|
||||
annotations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/fundamentals/alert-rules/annotation-label/#annotations
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/fundamentals/alert-rules/annotation-label/#annotations
|
||||
multi-dimensional-alerts:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/examples/multi-dimensional-alerts/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/examples/multi-dimensional-alerts/
|
||||
---
|
||||
|
||||
# Alerting best practices
|
||||
|
||||
Designing and configuring an effective alerting system takes time. This guide focuses on building alerting systems that scale with real-world operations.
|
||||
|
||||
The practices described here are intentionally high-level and apply regardless of tooling. Whether you use Prometheus, Grafana Alerting, or another stack, the same constraints apply: complex systems, imperfect signals, and humans on call.
|
||||
|
||||
Alerting is never finished. It evolves with incidents, organizational changes, and the systems it’s meant to protect.
|
||||
|
||||
{{< shared id="alert-planning-fundamentals" >}}
|
||||
|
||||
## Prioritize symptoms, but don’t ignore infrastructure signals
|
||||
|
||||
Alerts should primarily detect user-facing failures, not internal component behavior. Users don't care that a pod restarted; they care when the application is slow or failing. Symptom-based alerts tie directly to user impact.
|
||||
|
||||
Reliability metrics that impact users—latency, errors, availability—are better paging signals than infrastructure events or internal errors.
|
||||
|
||||
That said, infrastructure signals still matter. They can act as early warning indicators and are often useful when alerting maturity is low. A sustained spike in CPU or memory usage might not justify a page, but it can help explain or anticipate symptom-based failures.
|
||||
|
||||
Infrastructure alerts tend to be noisy and are often ignored when treated like paging signals. They are usually better suited for lower-severity channels such as dashboards, alert lists, or non-paging destinations like a dedicated Slack channel, where they can be monitored without interrupting on-call.
|
||||
|
||||
The key is balance as your alerting matures. Use infrastructure alerts to support diagnosis and prevention, not as a replacement for symptom-based alerts.
|
||||
|
||||
## Escalate priority based on confidence
|
||||
|
||||
Alert priority is often tied to user impact and the urgency to respond, but confidence should determine when escalation is necessary.
|
||||
|
||||
In this context, escalation defines how responders are notified as confidence grows. This can include increasing alert priority, widening notification, paging additional responders, or opening an incident once intervention is clearly required.
|
||||
|
||||
Early signals are often ambiguous, and confidence in a non-transient failure is usually low. Paging too early creates noise; paging too late means users are impacted for longer before anyone acts. A small or sudden increase in latency may not justify immediate action, but it can indicate a failure in progress.
|
||||
|
||||
Confidence increases as signals become stronger or begin to correlate.
|
||||
|
||||
Escalation is justified when issues are sustained or reinforced by multiple signals. For example, high latency combined with a rising error rate, or the same event firing over a sustained period. These patterns reduce the chance of transient noise and increase the likelihood of real impact.
|
||||
|
||||
Use confidence in user impact to drive escalation and avoid unnecessary pages.
|
||||
|
||||
## Scope alerts for scalability and actionability
|
||||
|
||||
In distributed systems, avoid creating separate alert rules for every host, service, or endpoint. Instead, define alert rules that scale automatically using [multi-dimensional alert rules](ref:multi-dimensional-alerts). This reduces rule duplication and allows alerting to scale as the system grows.
|
||||
|
||||
Start simple. Default to a single dimension such as `service` or `endpoint` to keep alerts manageable. Add dimensions only when they improve actionability. For example, when missing a dimension like `region` hides failures or doesn't provide enough information to act quickly.
|
||||
|
||||
Additional dimensions like `region` or `instance` can help identify the root cause, but more isn't always better.
|
||||
|
||||
## Design alerts for first responders and clear actions
|
||||
|
||||
Alerts should be designed for the first responder, not the person who created the alert. Anyone on call should be able to understand what's wrong and what to do next without deep knowledge of the system or alert configuration.
|
||||
|
||||
Avoid vague alerts that force responders to spend time figuring out context. Every alert should clearly explain why it exists, what triggered it, and how to investigate. Use [annotations](ref:annotations) to link to relevant dashboards and runbooks, which are essential for faster resolution.
|
||||
|
||||
Alerts should indicate a real problem and be actionable, even if the impact is low. Informational alerts add noise without improving reliability.
|
||||
|
||||
If no action is possible, it shouldn't be an alert—consider using a dashboard instead. Over time, alerts behave like technical debt: easy to create, costly to maintain, and hard to remove.
|
||||
|
||||
Review alerts often and remove those that don’t lead to action.
|
||||
|
||||
## Alerts should have an owner and system scope
|
||||
|
||||
Alerts without ownership are often ignored. Every alert must have an owner: a team responsible for maintaining the alert and responding when it fires.
|
||||
|
||||
Alerts must also define a system scope, such as a service or infrastructure component. Scope provides organizational context and connects alerts with ownership. Defining clear scopes is easier when services are treated as first-class entities, and organizations are built around service ownership.
|
||||
|
||||
> [Service Center in Grafana Cloud](/docs/grafana-cloud/alerting-and-irm/service-center/) can help operate a service-oriented view of your system and align alert scope with ownership.
|
||||
|
||||
After scope, ownership, and alert priority are defined, routing determines where alerts go and how they escalate. **Notification routing is as important as the alerts**.
|
||||
|
||||
Alerts should be delivered to the right team and channel based on priority, ownership, and team workflows. Use [notification policies](ref:notification-policies) to define a routing tree that matches the context of your service or scope:
|
||||
|
||||
- Define a parent policy for default routing within the scope.
|
||||
- Define nested policies for specific cases or higher-priority issues.
|
||||
|
||||
## Prevent notification overload with alert grouping
|
||||
|
||||
Without alert grouping, responders can receive many notifications for the same underlying problem.
|
||||
|
||||
For example, a database failure can trigger several alerts at the same time like increased latency, higher error rates, and internal errors. Paging separately for each symptom quickly turns into notification spam, even though there is a single root cause.
|
||||
|
||||
[Notification grouping](ref:group-alert-notifications) consolidates related alerts into a single notification. Instead of receiving multiple pages for the same issue, responders get one alert that represents the incident and includes all related firing alerts.
|
||||
|
||||
Grouping should follow operational boundaries such as service or owner, as defined by notification policies. Downstream or cascading failures should be grouped together so they surface as one issue rather than many.
|
||||
|
||||
## Mitigate flapping alerts
|
||||
|
||||
Short-lived failure spikes often trigger alerts that auto-resolve quickly. Alerting on transient failures creates noise and leads responders to ignore them.
|
||||
|
||||
Require issues to persist before alerting. Set a [pending period](ref:pending-period) to define how long a condition must remain true before firing. For example, instead of alerting immediately on high error rate, require it to stay above the threshold for some minutes.
|
||||
|
||||
Also, stabilize alerts by tuning query ranges and aggregations. Using raw data makes alerts sensitive to noise. Instead, evaluate over a time window and aggregate the data to smooth short spikes.
|
||||
|
||||
```promql
|
||||
# Reacts to transient spikes. Avoid this.
|
||||
cpu_usage > 90
|
||||
|
||||
# Smooth fluctuations.
|
||||
avg_over_time(cpu_usage[5m]) > 90
|
||||
```
|
||||
|
||||
For latency and error-based alerts, percentiles are often more useful than averages:
|
||||
|
||||
```promql
|
||||
quantile_over_time(0.95, http_duration_seconds[5m]) > 3
|
||||
```
|
||||
|
||||
Finally, avoid rapid resolve-and-fire notifications by using [`keep_firing_for`](ref:keep-firing-for) or [recovery thresholds](ref:recovery-threshold) to keep alerts active briefly during recovery. Both options reduce flapping and unnecessary notifications.
|
||||
|
||||
## Graduate symptom-based alerts into SLOs
|
||||
|
||||
When a symptom-based alert fires frequently, it usually indicates a reliability concern that should be measured and managed more deliberately. This is often a sign that the alert could evolve into an [SLO](/docs/grafana-cloud/alerting-and-irm/slo/).
|
||||
|
||||
Traditional alerts create pressure to react immediately, while error budgets introduce a buffer of time to act, changing how urgency is handled. Alerts can then be defined in terms of error budget burn rate rather than reacting to every minor deviation.
|
||||
|
||||
SLOs also align distinct teams around common reliability goals by providing a shared definition of what "good" looks like. They help consolidate multiple symptom alerts into a single user-facing objective.
|
||||
|
||||
For example, instead of several teams alerting on high latency, a single SLO can be used across teams to capture overall API performance.
|
||||
|
||||
## Integrate alerting into incident post-mortems
|
||||
|
||||
Every incident is an opportunity to improve alerting. After each incident, evaluate whether alerts helped responders act quickly or added unnecessary noise.
|
||||
|
||||
Assess which alerts fired, and how they influenced incident response. Review whether alerts triggered too late, too early, or without enough context, and adjust thresholds, priority, or escalation based on what actually happened.
|
||||
|
||||
Use [silences](ref:silences) during active incidents to reduce repeated notifications, but scope them carefully to avoid silencing unrelated alerts.
|
||||
|
||||
Post-mortems should evaluate alerts with root causes and lessons learned. If responders lacked key information during the incident, enrich alerts with additional context, dashboards, or better guidance.
|
||||
|
||||
## Alerts should be continuously improved
|
||||
|
||||
Alerting is an iterative process. Alerts that aren’t reviewed and refined lose effectiveness as systems and traffic patterns change.
|
||||
|
||||
Schedule regular reviews of existing alerts. Remove alerts that don’t lead to action, and tune alerts or thresholds that fire too often without providing useful signal. Reduce false positives to combat alert fatigue.
|
||||
|
||||
Prioritize clarity and simplicity in alert design. Simpler alerts are easier to understand, maintain, and trust under pressure. Favor fewer high-quality, actionable alerts over a large number of low-value ones.
|
||||
|
||||
Use dashboards and observability tools for investigation, not alerts.
|
||||
|
||||
{{< /shared >}}
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/connectivity-errors/
|
||||
aliases:
|
||||
- ../best-practices/connectivity-errors/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/connectivity-errors/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/guides/connectivity-errors/
|
||||
description: Learn how to detect and handle connectivity issues in alerts using Prometheus, Grafana Alerting, or both.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -14,7 +16,7 @@ labels:
|
||||
- oss
|
||||
menuTitle: Handle connectivity errors
|
||||
title: Handle connectivity errors in alerts
|
||||
weight: 1010
|
||||
weight: 1020
|
||||
refs:
|
||||
pending-period:
|
||||
- pattern: /docs/grafana/
|
||||
+4
-2
@@ -1,5 +1,7 @@
|
||||
---
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/best-practices/missing-data/
|
||||
aliases:
|
||||
- ../best-practices/missing-data/ # /docs/grafana/<GRAFANA_VERSION>/alerting/best-practices/missing-data/
|
||||
canonical: https://grafana.com/docs/grafana/latest/alerting/guides/missing-data/
|
||||
description: Learn how to detect missing metrics and design alerts that handle gaps in data in Prometheus and Grafana Alerting.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -14,7 +16,7 @@ labels:
|
||||
- oss
|
||||
menuTitle: Handle missing data
|
||||
title: Handle missing data in Grafana Alerting
|
||||
weight: 1020
|
||||
weight: 1030
|
||||
refs:
|
||||
connectivity-errors-guide:
|
||||
- pattern: /docs/grafana/
|
||||
@@ -41,9 +41,13 @@ Select a group to expand it and view the list of alert rules within that group.
|
||||
|
||||
The list view includes a number of filters to simplify managing large volumes of alerts.
|
||||
|
||||
## Filter and save searches
|
||||
|
||||
Click the **Filter** button to open the filter popup. You can filter by name, label, folder/namespace, evaluation group, data source, contact point, rule source, rule state, rule type, and the health of the alert rule from the popup menu. Click **Apply** at the bottom of the filter popup to enact the filters as you search.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-list-view-filter.png" max-width="750px" alt="Alert rule filter options" >}}
|
||||
Click the **Saved searches** button to open the list of previously saved searches, or click **+ Save current search** to add your current search to the saved searches list. You can also rename a saved search or set it as a default search. When you set a saved search as the default search, the Alert rules page opens with the search applied.
|
||||
|
||||
{{< figure src="/media/docs/alerting/alerting-saved-searches.png" max-width="750px" alt="Alert rule filter options" >}}
|
||||
|
||||
## Change alert rules list view
|
||||
|
||||
|
||||
@@ -128,35 +128,48 @@ The set up process verifies the path and provides an error message if a problem
|
||||
|
||||
#### Synchronization limitations
|
||||
|
||||
Full instance sync is not available in Grafana Cloud.
|
||||
{{< admonition type="caution" >}}
|
||||
|
||||
In Grafana OSS/Enterprise:
|
||||
Full instance sync is not available in Grafana Cloud and is experimental and unsupported in Grafana OSS/Enterprise.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
To have access to full instance sync you must explicitly enable the option.
|
||||
|
||||
The following applies:
|
||||
|
||||
- If you try to perform a full instance sync with resources that contain alerts or panels, the connection will be blocked.
|
||||
- You won't be able to create new alerts or library panels after setup is completed.
|
||||
- If you opted for full instance sync and want to use alerts and library panels, you'll have to delete the provisioned repository and connect again with folder sync.
|
||||
|
||||
#### Set up synchronization
|
||||
|
||||
Choose to either sync your entire organization resources with external storage, or to sync certain resources to a new Grafana folder (with up to 10 connections).
|
||||
You can sync external resources into a new folder without affecting the rest of your instance.
|
||||
|
||||
- Choose **Sync all resources with external storage** if you want to sync and manage your entire Grafana instance through external storage. With this option, all of your dashboards are synced to that one repository. You can only have one provisioned connection with this selection, and you won't have the option of setting up additional repositories to connect to.
|
||||
To set up synchronization:
|
||||
|
||||
- Choose **Sync external storage to new Grafana folder** to sync external resources into a new folder without affecting the rest of your instance. You can repeat this process for up to 10 connections.
|
||||
1. Select which resources you want to sync.
|
||||
|
||||
Next, enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI.
|
||||
1. Enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI.
|
||||
|
||||
Click **Synchronize** to continue.
|
||||
1. Click **Synchronize** to continue.
|
||||
|
||||
1. You can repeat this process for up to 10 connections.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
|
||||
Optionally, you can export any unmanaged resources into the provisioned folder. See how in [Synchronize with external storage](#synchronize-with-external-storage).
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
### Synchronize with external storage
|
||||
|
||||
After this one time step, all future updates are automatically saved to the local file path and provisioned back to the instance.
|
||||
In this step you proceed to synchronize the resources selected in the previous step. Optionally, you can check the **Migrate existing resources** box to migrate your unmanaged dashboards to the provisioned folder.
|
||||
|
||||
During the initial synchronization, your dashboards will be temporarily unavailable. No data or configurations will be lost.
|
||||
Select **Begin synchronization** to start the process. After this one time step, all future updates are automatically saved to the local file path and provisioned back to the instance.
|
||||
|
||||
Note that during the initial synchronization, your dashboards will be temporarily unavailable. No data or configurations will be lost.
|
||||
How long the process takes depends upon the number of resources involved.
|
||||
|
||||
Select **Begin synchronization** to start the process.
|
||||
|
||||
### Choose additional settings
|
||||
|
||||
If you wish, you can make any files synchronized as as **Read only** so no changes can be made to the resources through Grafana.
|
||||
|
||||
@@ -132,17 +132,35 @@ To connect your GitHub repository:
|
||||
|
||||
### Choose what to synchronize
|
||||
|
||||
In this step, you can decide which elements to synchronize. The available options depend on the status of your Grafana instance:
|
||||
|
||||
- If the instance contains resources in an incompatible data format, you'll have to migrate all the data using instance sync. Folder sync won't be supported.
|
||||
- If there's already another connection using folder sync, instance sync won't be offered.
|
||||
You can sync external resources into a new folder without affecting the rest of your instance.
|
||||
|
||||
To set up synchronization:
|
||||
|
||||
- Choose **Sync all resources with external storage** if you want to sync and manage your entire Grafana instance through external storage. With this option, all of your dashboards are synced to that one repository. You can only have one provisioned connection with this selection, and you won't have the option of setting up additional repositories to connect to.
|
||||
- Choose **Sync external storage to new Grafana folder** to sync external resources into a new folder without affecting the rest of your instance. You can repeat this process for up to 10 connections.
|
||||
1. Select which resources you want to sync.
|
||||
|
||||
Next, enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI. Click **Synchronize** to continue.
|
||||
1. Enter a **Display name** for the repository connection. Resources stored in this connection appear under the chosen display name in the Grafana UI.
|
||||
|
||||
1. Click **Synchronize** to continue.
|
||||
|
||||
1. You can repeat this process for up to 10 connections.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
|
||||
Optionally, you can export any unmanaged resources into the provisioned folder. See how in [Synchronize with external storage](#synchronize-with-external-storage).
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
#### Full instance sync
|
||||
|
||||
Full instance sync is not available in Grafana Cloud and is experimental and unsupported in Grafana OSS/Enterprise.
|
||||
|
||||
To have access to this option you must enable experimental instance sync on purpose.
|
||||
|
||||
### Synchronize with external storage
|
||||
|
||||
After this one time step, all future updates are automatically saved to the Git repository and provisioned back to the instance.
|
||||
|
||||
Check the **Migrate existing resources** box to migrate your unmanaged dashboards to the provisioned folder.
|
||||
|
||||
### Choose additional settings
|
||||
|
||||
|
||||
@@ -47,7 +47,7 @@ Using Git Sync, you can:
|
||||
|
||||
{{< admonition type="caution" >}}
|
||||
|
||||
Git Sync only works with specific folders for the moment. Full-instance sync is not currently supported.
|
||||
Full instance sync is not available in Grafana Cloud and is experimental and unsupported in Grafana OSS/Enterprise.
|
||||
|
||||
{{< /admonition >}}
|
||||
|
||||
@@ -84,7 +84,7 @@ Refer to [Requirements](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/obser
|
||||
|
||||
- You can only sync dashboards and folders. Refer to [Supported resources](#supported-resources) for more information.
|
||||
- If you're using Git Sync in Grafana OSS and Grafana Enterprise, some resources might be in an incompatible data format and won't be synced.
|
||||
- Full-instance sync is not available in Grafana Cloud and has limitations in Grafana OSS and Grafana Enterprise. Refer to [Choose what to synchronize](../git-sync-setup/#choose-what-to-synchronize) for more details.
|
||||
- Full-instance sync is not available in Grafana Cloud and is experimental in Grafana OSS and Grafana Enterprise. Refer to [Choose what to synchronize](../git-sync-setup/#choose-what-to-synchronize) for more details.
|
||||
- When migrating to full instance sync, during the synchronization process your resources will be temporarily unavailable. No one will be able to create, edit, or delete resources during this process.
|
||||
- If you want to manage existing resources with Git Sync, you need to save them as JSON files and commit them to the synced repository. Open a PR to import, copy, move, or save a dashboard.
|
||||
- Restoring resources from the UI is currently not possible. As an alternative, you can restore dashboards directly in your GitHub repository by raising a PR, and they will be updated in Grafana.
|
||||
|
||||
@@ -112,6 +112,12 @@ For example, this video demonstrates the visual Prometheus query builder:
|
||||
|
||||
For general information about querying in Grafana, and common options and user interface elements across all query editors, refer to [Query and transform data](ref:query-transform-data).
|
||||
|
||||
## Build a dashboard from the data source
|
||||
|
||||
After you've configured a data source, you can start creating a dashboard directly from it, by clicking the **Build a dashboard** button.
|
||||
|
||||
For more information, refer to [Begin dashboard creation from data source configuration](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/dashboards/build-dashboards/create-dashboard/#begin-dashboard-creation-from-connections).
|
||||
|
||||
## Special data sources
|
||||
|
||||
Grafana includes three special data sources:
|
||||
|
||||
@@ -105,6 +105,11 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/visualizations/
|
||||
cloudwatch-troubleshooting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/troubleshooting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/troubleshooting/
|
||||
---
|
||||
|
||||
# Amazon CloudWatch data source
|
||||
@@ -119,6 +124,7 @@ The following documents will help you get started working with the CloudWatch da
|
||||
- [CloudWatch query editor](ref:cloudwatch-query-editor)
|
||||
- [Templates and variables](ref:cloudwatch-template-variables)
|
||||
- [Configure AWS authentication](ref:cloudwatch-aws-authentication)
|
||||
- [Troubleshoot CloudWatch issues](ref:cloudwatch-troubleshooting)
|
||||
|
||||
## Import pre-configured dashboards
|
||||
|
||||
|
||||
@@ -1,11 +1,12 @@
|
||||
---
|
||||
aliases:
|
||||
- ../data-sources/aws-CloudWatch/
|
||||
- ../data-sources/aws-CloudWatch/preconfig-CloudWatch-dashboards/
|
||||
- ../data-sources/aws-CloudWatch/provision-CloudWatch/
|
||||
- CloudWatch/
|
||||
- preconfig-CloudWatch-dashboards/
|
||||
- provision-CloudWatch/
|
||||
- ../../data-sources/aws-cloudwatch/configure/
|
||||
- ../../data-sources/aws-cloudwatch/
|
||||
- ../../data-sources/aws-cloudwatch/preconfig-cloudwatch-dashboards/
|
||||
- ../../data-sources/aws-cloudwatch/provision-cloudwatch/
|
||||
- ../cloudwatch/
|
||||
- ../preconfig-cloudwatch-dashboards/
|
||||
- ../provision-cloudwatch/
|
||||
description: This document provides configuration instructions for the CloudWatch data source.
|
||||
keywords:
|
||||
- grafana
|
||||
@@ -25,11 +26,6 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
provisioning-data-sources:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
@@ -40,26 +36,16 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#aws
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-grafana/#aws
|
||||
alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/
|
||||
build-dashboards:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/
|
||||
data-source-management:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
CloudWatch-aws-authentication:
|
||||
cloudwatch-aws-authentication:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-CloudWatch/aws-authentication/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-CloudWatch/aws-authentication/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/
|
||||
private-data-source-connect:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
@@ -108,7 +94,7 @@ The following are configuration options for the CloudWatch data source.
|
||||
Grafana plugin requests to AWS are made on behalf of an AWS Identity and Access Management (IAM) role or IAM user.
|
||||
The IAM user or IAM role must have the associated policies to perform certain API actions.
|
||||
|
||||
For authentication options and configuration details, refer to [AWS authentication](aws-authentication/).
|
||||
For authentication options and configuration details, refer to [AWS authentication](ref:cloudwatch-aws-authentication).
|
||||
|
||||
| Setting | Description |
|
||||
| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
@@ -153,7 +139,7 @@ You must use both an access key ID and a secret access key to authenticate.
|
||||
|
||||
Grafana automatically creates a link to a trace in X-Ray data source if logs contain the `@xrayTraceId` field. To use this feature, you must already have an X-Ray data source configured. For details, see the [X-Ray data source docs](/grafana/plugins/grafana-X-Ray-datasource/). To view the X-Ray link, select the log row in either the Explore view or dashboard [Logs panel](ref:logs) to view the log details section.
|
||||
|
||||
To log the `@xrayTraceId`, refer to the [AWS X-Ray documentation](https://docs.amazonaws.cn/en_us/xray/latest/devguide/xray-services.html). To provide the field to Grafana, your log queries must also contain the `@xrayTraceId` field, for example by using the query `fields @message, @xrayTraceId`.
|
||||
To log the `@xrayTraceId`, refer to the [AWS X-Ray documentation](https://docs.aws.amazon.com/xray/latest/devguide/xray-services.html). To provide the field to Grafana, your log queries must also contain the `@xrayTraceId` field, for example by using the query `fields @message, @xrayTraceId`.
|
||||
|
||||
**Private data source connect** - _Only for Grafana Cloud users._
|
||||
|
||||
@@ -172,7 +158,7 @@ To troubleshoot issues while setting up the CloudWatch data source, check the `/
|
||||
### IAM policy examples
|
||||
|
||||
To read CloudWatch metrics and EC2 tags, instances, regions, and alarms, you must grant Grafana permissions via IAM.
|
||||
You can attach these permissions to the IAM role or IAM user you configured in [AWS authentication](aws-authentication/).
|
||||
You can attach these permissions to the IAM role or IAM user you configured in [AWS authentication](ref:cloudwatch-aws-authentication).
|
||||
|
||||
**Metrics-only permissions:**
|
||||
|
||||
@@ -323,7 +309,7 @@ You can attach these permissions to the IAM role or IAM user you configured in [
|
||||
Cross-account observability lets you retrieve metrics and logs across different accounts in a single region, but you can't query EC2 Instance Attributes across accounts because those come from the EC2 API and not the CloudWatch API.
|
||||
{{< /admonition >}}
|
||||
|
||||
For more information on configuring authentication, refer to [Configure AWS authentication](ref:CloudWatch-aws-authentication).
|
||||
For more information on configuring authentication, refer to [Configure AWS authentication](ref:cloudwatch-aws-authentication).
|
||||
|
||||
### CloudWatch Logs data protection
|
||||
|
||||
|
||||
@@ -34,11 +34,6 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/#navigate-the-query-tab
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/#navigate-the-query-tab
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/
|
||||
@@ -183,7 +178,7 @@ If you use the expression field to reference another query, such as `queryA * 2`
|
||||
When you select `Builder` mode within the Metric search editor, a new Account field is displayed. Use the `Account` field to specify which of the linked monitoring accounts to target for the given query. By default, the `All` option is specified, which will target all linked accounts.
|
||||
|
||||
While in `Code` mode, you can specify any math expression. If the Monitoring account badge displays in the query editor header, all `SEARCH` expressions entered in this field will be cross-account by default and can query metrics from linked accounts. Note that while queries run cross-account, the autocomplete feature currently doesn't fetch cross-account resources, so you'll need to manually specify resource names when writing cross-account queries.
|
||||
You can limit the search to one or a set of accounts, as documented in the [AWS documentation](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html).
|
||||
You can limit the search to one or a set of accounts, as documented in the [AWS documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html).
|
||||
|
||||
### Period macro
|
||||
|
||||
@@ -198,7 +193,7 @@ The link provided is valid for any account but displays the expected metrics onl
|
||||
|
||||
{{< figure src="/media/docs/cloudwatch/cloudwatch-deep-link-v12.1.png" caption="CloudWatch deep linking" >}}
|
||||
|
||||
This feature is not available for metrics based on [metric math expressions](#metric-math-expressions).
|
||||
This feature is not available for metrics based on [metric math expressions](#use-metric-math-expressions).
|
||||
|
||||
### Use Metric Insights syntax
|
||||
|
||||
@@ -319,9 +314,9 @@ The CloudWatch plugin monitors and troubleshoots applications that span multiple
|
||||
|
||||
To enable cross-account observability, complete the following steps:
|
||||
|
||||
1. Go to the [Amazon CloudWatch documentation](http://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) and follow the instructions for enabling cross-account observability.
|
||||
1. Go to the [Amazon CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-Unified-Cross-Account.html) and follow the instructions for enabling cross-account observability.
|
||||
|
||||
1. Add [two API actions](https://grafana.com//docs/grafana/latest/datasources/aws-cloudwatch/configure/#cross-account-observability-permissions) to the IAM policy attached to the role/user running the plugin.
|
||||
1. Add [two API actions](https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/configure/#cross-account-observability-permissions) to the IAM policy attached to the role/user running the plugin.
|
||||
|
||||
Cross-account querying is available in the plugin through the **Logs**, **Metric search**, and **Metric Insights** modes.
|
||||
After you have configured it, you'll see a **Monitoring account** badge in the query editor header.
|
||||
|
||||
@@ -0,0 +1,519 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/aws-cloudwatch/troubleshooting/
|
||||
description: Troubleshooting guide for the Amazon CloudWatch data source in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- cloudwatch
|
||||
- aws
|
||||
- troubleshooting
|
||||
- errors
|
||||
- authentication
|
||||
- query
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Troubleshooting
|
||||
title: Troubleshoot Amazon CloudWatch data source issues
|
||||
weight: 500
|
||||
refs:
|
||||
configure-cloudwatch:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/configure/
|
||||
cloudwatch-aws-authentication:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/aws-authentication/
|
||||
cloudwatch-template-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/template-variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/template-variables/
|
||||
cloudwatch-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-cloudwatch/query-editor/
|
||||
private-data-source-connect:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
---
|
||||
|
||||
# Troubleshoot Amazon CloudWatch data source issues
|
||||
|
||||
This document provides solutions to common issues you may encounter when configuring or using the Amazon CloudWatch data source. For configuration instructions, refer to [Configure CloudWatch](ref:configure-cloudwatch).
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
The data source health check validates both metrics and logs permissions. If your IAM policy only grants access to one of these (for example, metrics-only or logs-only), the health check displays a red status. However, the service you have permissions for is still usable—you can query metrics or logs based on whichever permissions are configured.
|
||||
{{< /admonition >}}
|
||||
|
||||
## Authentication errors
|
||||
|
||||
These errors occur when AWS credentials are invalid, missing, or don't have the required permissions.
|
||||
|
||||
### "Access Denied" or "Not authorized to perform this operation"
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Save & test fails with "Access Denied"
|
||||
- Queries return authorization errors
|
||||
- Namespaces, metrics, or dimensions don't load
|
||||
|
||||
**Possible causes and solutions:**
|
||||
|
||||
| Cause | Solution |
|
||||
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| IAM policy missing required permissions | Attach the appropriate IAM policy to your user or role. For metrics, you need `cloudwatch:ListMetrics`, `cloudwatch:GetMetricData`, and related permissions. For logs, you need `logs:DescribeLogGroups`, `logs:StartQuery`, `logs:GetQueryResults`, and related permissions. Refer to [Configure CloudWatch](ref:configure-cloudwatch) for complete policy examples. |
|
||||
| Incorrect access key or secret key | Verify the credentials in the AWS Console under **IAM** > **Users** > your user > **Security credentials**. Generate new credentials if necessary. |
|
||||
| Credentials have expired | For temporary credentials, generate new ones. For access keys, verify they haven't been deactivated or deleted. |
|
||||
| Wrong AWS region | Verify the default region in the data source configuration matches where your resources are located. |
|
||||
| Assume Role ARN is incorrect | Verify the role ARN format: `arn:aws:iam::<account-id>:role/<role-name>`. Check that the role exists in the AWS Console. |
|
||||
|
||||
### "Unable to assume role"
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Authentication fails when using Assume Role ARN
|
||||
- Error message references STS or AssumeRole
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the trust relationship on the IAM role allows the Grafana credentials to assume it.
|
||||
1. Check the trust policy includes the correct principal (the user or role running Grafana).
|
||||
1. If using an external ID, ensure it matches exactly in both the role's trust policy and the Grafana data source configuration.
|
||||
1. Verify the base credentials have the `sts:AssumeRole` permission.
|
||||
1. Check that the role ARN is correct and the role exists.
|
||||
|
||||
**Example trust policy:**
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Principal": {
|
||||
"AWS": "arn:aws:iam::<your-account-id>:user/<grafana-user>"
|
||||
},
|
||||
"Action": "sts:AssumeRole",
|
||||
"Condition": {
|
||||
"StringEquals": {
|
||||
"sts:ExternalId": "<your-external-id>"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### AWS SDK Default authentication not working
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Data source test fails when using AWS SDK Default
|
||||
- Works locally but fails in production
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify AWS credentials are configured in the environment where Grafana runs.
|
||||
1. Check for credentials in the default locations:
|
||||
- Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`)
|
||||
- Shared credentials file (`~/.aws/credentials`)
|
||||
- EC2 instance metadata (if running on EC2)
|
||||
- ECS task role (if running in ECS)
|
||||
- EKS service account (if running in EKS)
|
||||
1. Ensure the Grafana process has permission to read the credentials file.
|
||||
1. For EKS with IRSA, set the pod's security context to allow user 472 (grafana) to access the projected token. Refer to [AWS authentication](ref:cloudwatch-aws-authentication) for details.
|
||||
|
||||
### Credentials file not found
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Error indicates credentials file cannot be read
|
||||
- Authentication fails with "Credentials file" option
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Create the credentials file at `~/.aws/credentials` for the user running the `grafana-server` service.
|
||||
1. Verify the file has correct permissions (`0644`).
|
||||
1. If the file exists but isn't working, move it to `/usr/share/grafana/` and set permissions to `0644`.
|
||||
1. Ensure the profile name in the data source configuration matches a profile in the credentials file.
|
||||
|
||||
## Connection errors
|
||||
|
||||
These errors occur when Grafana cannot reach AWS CloudWatch endpoints.
|
||||
|
||||
### "Request timed out" or connection failures
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Data source test times out
|
||||
- Queries fail with timeout errors
|
||||
- Intermittent connection issues
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify network connectivity from the Grafana server to AWS endpoints.
|
||||
1. Check firewall rules allow outbound HTTPS (port 443) to AWS services.
|
||||
1. If using a VPC, ensure proper NAT gateway or VPC endpoint configuration.
|
||||
1. For Grafana Cloud connecting to private resources, configure [Private data source connect](ref:private-data-source-connect).
|
||||
1. Check if the default region is correct—incorrect regions may cause longer timeouts.
|
||||
1. Increase the timeout settings if queries involve large data volumes.
|
||||
|
||||
### Custom endpoint configuration issues
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Connection fails when using a custom endpoint
|
||||
- Endpoint URL rejected
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the endpoint URL format is correct.
|
||||
1. Ensure the endpoint is accessible from the Grafana server.
|
||||
1. Check that the endpoint supports the required AWS APIs.
|
||||
1. For VPC endpoints, verify the endpoint policy allows the required actions.
|
||||
|
||||
## CloudWatch Metrics query errors
|
||||
|
||||
These errors occur when querying CloudWatch Metrics.
|
||||
|
||||
### "No data" or empty results
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Query executes without error but returns no data
|
||||
- Charts show "No data" message
|
||||
|
||||
**Possible causes and solutions:**
|
||||
|
||||
| Cause | Solution |
|
||||
| ------------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
||||
| Time range doesn't contain data | Expand the dashboard time range. CloudWatch metrics have different retention periods based on resolution. |
|
||||
| Wrong namespace or metric name | Verify the namespace (for example, `AWS/EC2`) and metric name (for example, `CPUUtilization`) are correct. |
|
||||
| Incorrect dimensions | Ensure dimension names and values match your AWS resources exactly. |
|
||||
| Match Exact enabled incorrectly | When Match Exact is enabled, all dimensions must be specified. Try disabling it to see if metrics appear. |
|
||||
| Period too large | Reduce the period setting or set it to "auto" to ensure data points are returned for your time range. |
|
||||
| Custom metrics not configured | Add custom metric namespaces in the data source configuration under **Namespaces of Custom Metrics**. |
|
||||
|
||||
### "Metric not found" or metrics don't appear in drop-down
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Expected metrics don't appear in the query editor
|
||||
- Metric drop-down is empty for a namespace
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the metric exists in the selected region.
|
||||
1. For custom metrics, add the namespace to **Namespaces of Custom Metrics** in the data source configuration.
|
||||
1. Check that the IAM policy includes `cloudwatch:ListMetrics` permission.
|
||||
1. CloudWatch limits `ListMetrics` to 500 results per page. To retrieve more metrics, increase the `list_metrics_page_limit` setting in the [Grafana configuration file](https://grafana.com/docs/grafana/latest/datasources/aws-cloudwatch/configure/#configure-the-data-source-with-grafanaini).
|
||||
1. Use the Query Inspector to verify the API request and response.
|
||||
|
||||
### Dimension values not loading
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Dimension value drop-down doesn't populate
|
||||
- Wildcard searches return no results
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the IAM policy includes `cloudwatch:ListMetrics` permission.
|
||||
1. Check that the namespace and metric are selected before dimension values can load.
|
||||
1. For EC2 dimensions, ensure `ec2:DescribeTags` and `ec2:DescribeInstances` permissions are granted.
|
||||
1. Dimension values require existing metrics—if no metrics match, no values appear.
|
||||
|
||||
### "Too many data points" or API throttling
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Queries fail with throttling errors
|
||||
- Performance degrades with multiple panels
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Increase the period setting to reduce the number of data points.
|
||||
1. Reduce the time range of your queries.
|
||||
1. Use fewer dimensions or wildcard queries per panel.
|
||||
1. Request a quota increase for `GetMetricData` requests per second in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/).
|
||||
1. Enable query caching in Grafana to reduce API calls.
|
||||
|
||||
### Metric math expression errors
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Expression returns errors
|
||||
- Referenced metrics not found
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify each referenced metric has a unique ID set.
|
||||
1. Check that metric IDs start with a lowercase letter and contain only letters, numbers, and underscores.
|
||||
1. Ensure all referenced metrics are in the same query.
|
||||
1. Verify the expression syntax follows [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) documentation.
|
||||
1. Metric math expressions can't be used with Grafana alerting if they reference other query rows.
|
||||
|
||||
## CloudWatch Logs query errors
|
||||
|
||||
These errors occur when querying CloudWatch Logs.
|
||||
|
||||
### "Query failed" or logs don't appear
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Log queries return errors
|
||||
- No log data is displayed
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify log group names are correct and exist in the selected region.
|
||||
1. Check the IAM policy includes `logs:StartQuery`, `logs:GetQueryResults`, and `logs:DescribeLogGroups` permissions.
|
||||
1. Ensure the time range contains log data.
|
||||
1. Verify the query syntax is valid. For CloudWatch Logs Insights QL, test the query in the AWS Console.
|
||||
1. Select the correct query language (Logs Insights QL, OpenSearch PPL, or OpenSearch SQL) based on your query syntax.
|
||||
|
||||
### Log query timeout
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Query runs for a long time then fails
|
||||
- Error mentions timeout
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Increase the **Query timeout result** setting in the data source configuration (default is 30 minutes).
|
||||
1. Narrow the time range to reduce the amount of data scanned.
|
||||
1. Add filters to your query to limit results.
|
||||
1. Break complex queries into smaller, more focused queries.
|
||||
1. For alerting, the timeout defined in the [Grafana configuration file](https://grafana.com/docs/grafana/latest/setup-grafana/configure-grafana/#unified_alerting) takes precedence.
|
||||
|
||||
### Log groups not appearing in selector
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Log group selector is empty
|
||||
- Can't find expected log groups
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the IAM policy includes `logs:DescribeLogGroups` permission.
|
||||
1. Check that log groups exist in the selected region.
|
||||
1. For cross-account observability, ensure proper IAM permissions for `oam:ListSinks` and `oam:ListAttachedLinks`.
|
||||
1. Use prefix search to filter log groups if you have many groups.
|
||||
1. Verify the selected account (for cross-account) contains the expected log groups.
|
||||
|
||||
### OpenSearch SQL query errors
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- OpenSearch SQL queries fail
|
||||
- Syntax errors with SQL queries
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Specify the log group identifier or ARN in the `FROM` clause:
|
||||
|
||||
```sql
|
||||
SELECT * FROM `log_group_name` WHERE `@message` LIKE '%error%'
|
||||
```
|
||||
|
||||
1. For multiple log groups, use the `logGroups` function:
|
||||
|
||||
```sql
|
||||
SELECT * FROM `logGroups(logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
|
||||
```
|
||||
|
||||
1. Amazon CloudWatch supports only a subset of OpenSearch SQL commands. Refer to the [CloudWatch Logs documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_AnalyzeLogData_Languages.html) for supported syntax.
|
||||
|
||||
## Template variable errors
|
||||
|
||||
These errors occur when using template variables with the CloudWatch data source.
|
||||
|
||||
### Variables return no values
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Variable drop-down is empty
|
||||
- Dashboard fails to load with variable errors
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the data source connection is working.
|
||||
1. Check that the IAM policy includes permissions for the variable query type:
|
||||
- **Regions:** No additional permissions needed.
|
||||
- **Namespaces:** No additional permissions needed.
|
||||
- **Metrics:** Requires `cloudwatch:ListMetrics`.
|
||||
- **Dimension Values:** Requires `cloudwatch:ListMetrics`.
|
||||
- **EC2 Instance Attributes:** Requires `ec2:DescribeInstances`.
|
||||
- **EBS Volume IDs:** Requires `ec2:DescribeVolumes`.
|
||||
- **Resource ARNs:** Requires `tag:GetResources`.
|
||||
- **Log Groups:** Requires `logs:DescribeLogGroups`.
|
||||
1. For dependent variables, ensure parent variables have valid selections.
|
||||
1. Verify the region is set correctly (use "default" for the data source's default region).
|
||||
|
||||
For more information on template variables, refer to [CloudWatch template variables](ref:cloudwatch-template-variables).
|
||||
|
||||
### Multi-value template variables cause query failures
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Queries fail when selecting multiple dimension values
|
||||
- Error about search expression limits
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Search expressions are limited to 1,024 characters. Reduce the number of selected values.
|
||||
1. Use the asterisk (`*`) wildcard instead of selecting "All" to query all metrics for a dimension.
|
||||
1. Multi-valued template variables are only supported for dimension values—not for Region, Namespace, or Metric Name.
|
||||
|
||||
## Cross-account observability errors
|
||||
|
||||
These errors occur when using CloudWatch cross-account observability features.
|
||||
|
||||
### Cross-account queries fail
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Can't query metrics or logs from linked accounts
|
||||
- Monitoring account badge doesn't appear
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify cross-account observability is configured in the AWS CloudWatch console.
|
||||
1. Add the required IAM permissions:
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Action": ["oam:ListSinks", "oam:ListAttachedLinks"],
|
||||
"Effect": "Allow",
|
||||
"Resource": "*"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
1. Check that the monitoring account and source accounts are properly linked in AWS.
|
||||
1. Cross-account observability works within a single region—verify all accounts are in the same region.
|
||||
1. EC2 Instance Attributes can't be queried across accounts because they use the EC2 API, not the CloudWatch API.
|
||||
|
||||
## Quota and pricing issues
|
||||
|
||||
These issues relate to AWS service quotas and cost management.
|
||||
|
||||
### API throttling errors
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- "Rate exceeded" errors
|
||||
- Dashboard panels intermittently fail to load
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Reduce the frequency of dashboard refreshes.
|
||||
1. Increase the period setting to reduce `GetMetricData` requests.
|
||||
1. Enable query caching in Grafana (available in Grafana Enterprise and Grafana Cloud).
|
||||
1. Request a quota increase in the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/).
|
||||
1. Consider consolidating similar queries using metric math.
|
||||
|
||||
### Unexpectedly high CloudWatch costs
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- AWS CloudWatch costs are higher than expected
|
||||
- Frequent API calls from Grafana
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. The `GetMetricData` API doesn't qualify for the CloudWatch API free tier.
|
||||
1. Reduce dashboard auto-refresh frequency.
|
||||
1. Increase the period setting to reduce data points returned.
|
||||
1. Use query caching to reduce repeated API calls.
|
||||
1. Review variable query settings—set variable refresh to "On dashboard load" instead of "On time range change."
|
||||
1. Avoid using wildcards in dimensions when possible, as they generate search expressions with multiple API calls.
|
||||
|
||||
## Other common issues
|
||||
|
||||
These issues don't produce specific error messages but are commonly encountered.
|
||||
|
||||
### Custom metrics don't appear
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Custom metrics from applications or agents don't show in the namespace drop-down
|
||||
- Only standard AWS namespaces are visible
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Add your custom metric namespace to the **Namespaces of Custom Metrics** field in the data source configuration.
|
||||
1. Separate multiple namespaces with commas (for example, `CWAgent,CustomNamespace`).
|
||||
1. Verify custom metrics have been published to CloudWatch in the selected region.
|
||||
|
||||
### Pre-configured dashboards not working
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Imported dashboards show no data
|
||||
- Dashboard variables don't load
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify the data source name in the dashboard matches your CloudWatch data source.
|
||||
1. Check that the dashboard's AWS region setting matches where your resources are located.
|
||||
1. Ensure the IAM policy grants access to the required services (EC2, Lambda, RDS, etc.).
|
||||
1. Verify resources exist and are emitting metrics in the selected region.
|
||||
|
||||
### X-Ray trace links not appearing
|
||||
|
||||
**Symptoms:**
|
||||
|
||||
- Log entries don't show X-Ray trace links
|
||||
- `@xrayTraceId` field not appearing
|
||||
|
||||
**Solutions:**
|
||||
|
||||
1. Verify an X-Ray data source is configured and linked in the CloudWatch data source settings.
|
||||
1. Ensure your logs contain the `@xrayTraceId` field.
|
||||
1. Update log queries to include `@xrayTraceId` in the fields, for example: `fields @message, @xrayTraceId`.
|
||||
1. Configure your application to log X-Ray trace IDs. Refer to the [AWS X-Ray documentation](https://docs.aws.amazon.com/xray/latest/devguide/xray-services.html).
|
||||
|
||||
## Enable debug logging
|
||||
|
||||
To capture detailed error information for troubleshooting:
|
||||
|
||||
1. Set the Grafana log level to `debug` in the configuration file:
|
||||
|
||||
```ini
|
||||
[log]
|
||||
level = debug
|
||||
```
|
||||
|
||||
1. Review logs in `/var/log/grafana/grafana.log` (or your configured log location).
|
||||
1. Look for CloudWatch-specific entries that include request and response details.
|
||||
1. Reset the log level to `info` after troubleshooting to avoid excessive log volume.
|
||||
|
||||
## Get additional help
|
||||
|
||||
If you've tried the solutions above and still encounter issues:
|
||||
|
||||
1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues.
|
||||
1. Review the [CloudWatch plugin GitHub issues](https://github.com/grafana/grafana/issues) for known bugs.
|
||||
1. Consult the [AWS CloudWatch documentation](https://docs.aws.amazon.com/cloudwatch/) for service-specific guidance.
|
||||
1. Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud Contracted user.
|
||||
1. When reporting issues, include:
|
||||
- Grafana version
|
||||
- AWS region
|
||||
- Error messages (redact sensitive information)
|
||||
- Steps to reproduce
|
||||
- Query configuration (redact credentials and account IDs)
|
||||
@@ -111,3 +111,4 @@ After installing and configuring the Graphite data source you can:
|
||||
- Add [transformations](ref:transformations)
|
||||
- Add [annotations](ref:annotate-visualizations)
|
||||
- Set up [alerting](ref:alerting)
|
||||
- [Troubleshoot](troubleshooting/) common issues with the Graphite data source
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
---
|
||||
description: Troubleshoot common issues with the Graphite data source.
|
||||
keywords:
|
||||
- grafana
|
||||
- graphite
|
||||
- troubleshooting
|
||||
- guide
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Troubleshooting
|
||||
title: Troubleshoot Graphite data source issues
|
||||
weight: 400
|
||||
refs:
|
||||
configure-graphite:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/graphite/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/graphite/configure/
|
||||
query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/graphite/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/graphite/query-editor/
|
||||
---
|
||||
|
||||
# Troubleshoot Graphite data source issues
|
||||
|
||||
This document provides solutions for common issues you might encounter when using the Graphite data source.
|
||||
|
||||
## Connection issues
|
||||
|
||||
Use the following troubleshooting steps to resolve connection problems between Grafana and your Graphite server.
|
||||
|
||||
**Data source test fails with "Unable to connect":**
|
||||
|
||||
If the data source test fails, verify the following:
|
||||
|
||||
- The URL in your data source configuration is correct and accessible from the Grafana server.
|
||||
- The Graphite server is running and accepting connections.
|
||||
- Any firewall rules or network policies allow traffic between Grafana and the Graphite server.
|
||||
- If using TLS, ensure your certificates are valid and properly configured.
|
||||
|
||||
To test connectivity, run the following command from the Grafana server:
|
||||
|
||||
```sh
|
||||
curl -v <GRAPHITE_URL>/render
|
||||
```
|
||||
|
||||
Replace _`<GRAPHITE_URL>`_ with your Graphite server URL. A successful connection returns a response from the Graphite server.
|
||||
|
||||
**Authentication errors:**
|
||||
|
||||
If you receive 401 or 403 errors:
|
||||
|
||||
- Verify your Basic Auth username and password are correct.
|
||||
- Ensure the **With Credentials** toggle is enabled if your Graphite server requires cookies for authentication.
|
||||
- Check that your TLS client certificates are valid and match what the server expects.
|
||||
|
||||
For detailed authentication configuration, refer to [Configure the Graphite data source](ref:configure-graphite).
|
||||
|
||||
## Query issues
|
||||
|
||||
Use the following troubleshooting steps to resolve problems with Graphite queries.
|
||||
|
||||
**No data returned:**
|
||||
|
||||
If your query returns no data:
|
||||
|
||||
- Verify the metric path exists in your Graphite server by testing directly in the Graphite web interface.
|
||||
- Check that the time range in Grafana matches when data was collected.
|
||||
- Ensure wildcards in your query match existing metrics.
|
||||
- Confirm your query syntax is correct for your Graphite version.
|
||||
|
||||
**HTTP 500 errors with HTML content:**
|
||||
|
||||
Graphite-web versions before 1.6 return HTTP 500 errors with full HTML stack traces when a query fails. If you see error messages containing HTML tags:
|
||||
|
||||
- Check the Graphite server logs for the full error details.
|
||||
- Verify your query syntax is valid.
|
||||
- Ensure the requested time range doesn't exceed your Graphite server's capabilities.
|
||||
- Check that all functions used in your query are supported by your Graphite version.
|
||||
|
||||
**Parser errors in the query editor:**
|
||||
|
||||
If the query editor displays parser errors:
|
||||
|
||||
- Check for unbalanced parentheses in function calls.
|
||||
- Verify that function arguments are in the correct format.
|
||||
- Ensure metric paths don't contain unsupported characters.
|
||||
|
||||
For query syntax help, refer to [Graphite query editor](ref:query-editor).
|
||||
|
||||
## Version and feature issues
|
||||
|
||||
Use the following troubleshooting steps to resolve problems related to Graphite versions and features.
|
||||
|
||||
**Functions missing from the query editor:**
|
||||
|
||||
If expected functions don't appear in the query editor:
|
||||
|
||||
- Verify the correct Graphite version is selected in the data source configuration.
|
||||
- The available functions depend on the configured version. For example, tag-based functions require Graphite 1.1 or later.
|
||||
- If using a custom Graphite installation with additional functions, ensure the version setting matches your server.
|
||||
|
||||
**Tag-based queries not working:**
|
||||
|
||||
If `seriesByTag()` or other tag functions fail:
|
||||
|
||||
- Confirm your Graphite server is version 1.1 or later.
|
||||
- Verify the Graphite version setting in your data source configuration matches your actual server version.
|
||||
- Check that tags are properly configured in your Graphite server.
|
||||
|
||||
## Performance issues
|
||||
|
||||
Use the following troubleshooting steps to address slow queries or timeouts.
|
||||
|
||||
**Queries timing out:**
|
||||
|
||||
If queries consistently time out:
|
||||
|
||||
- Increase the **Timeout** setting in the data source configuration.
|
||||
- Reduce the time range of your query.
|
||||
- Use more specific metric paths instead of broad wildcards.
|
||||
- Consider using `summarize()` or `consolidateBy()` functions to reduce the amount of data returned.
|
||||
- Check your Graphite server's performance and resource utilization.
|
||||
|
||||
**Slow autocomplete in the query editor:**
|
||||
|
||||
If metric path autocomplete is slow:
|
||||
|
||||
- This often indicates a large number of metrics in your Graphite server.
|
||||
- Use more specific path prefixes to narrow the search scope.
|
||||
- Check your Graphite server's index performance.
|
||||
|
||||
## MetricTank-specific issues
|
||||
|
||||
If you're using MetricTank as your Graphite backend, use the following troubleshooting steps.
|
||||
|
||||
**Rollup indicator not appearing:**
|
||||
|
||||
If the rollup indicator doesn't display when expected:
|
||||
|
||||
- Verify **Metrictank** is selected as the Graphite backend type in the data source configuration.
|
||||
- Ensure the **Rollup indicator** toggle is enabled.
|
||||
- The indicator only appears when data aggregation actually occurs.
|
||||
|
||||
**Unexpected data aggregation:**
|
||||
|
||||
If you see unexpected aggregation in your data:
|
||||
|
||||
- Check the rollup configuration in your MetricTank instance.
|
||||
- Adjust the time range or use `consolidateBy()` to control aggregation behavior.
|
||||
- Review the query processing metadata in the panel inspector for details on how data was processed.
|
||||
|
||||
## Get additional help
|
||||
|
||||
If you continue to experience issues:
|
||||
|
||||
- Check the [Grafana community forums](https://community.grafana.com/) for similar issues and solutions.
|
||||
- Review the [Graphite documentation](https://graphite.readthedocs.io/) for additional configuration options.
|
||||
- Contact [Grafana Support](https://grafana.com/support/) if you're an Enterprise, Cloud Pro, or Cloud Advanced customer.
|
||||
|
||||
When reporting issues, include the following information:
|
||||
|
||||
- Grafana version
|
||||
- Graphite version (for example, 1.1.x) and backend type (Default or MetricTank)
|
||||
- Authentication method (Basic Auth, TLS, or none)
|
||||
- Error messages (redact sensitive information)
|
||||
- Steps to reproduce the issue
|
||||
- Relevant configuration such as data source settings, timeout values, and Graphite version setting (redact passwords and other credentials)
|
||||
- Sample query (if applicable, with sensitive data redacted)
|
||||
@@ -369,7 +369,7 @@
|
||||
"description": "For data source plugins. Proxy routes used for plugin authentication and adding headers to HTTP requests made by the plugin. For more information, refer to [Authentication for data source plugins](https://grafana.com/developers/plugin-tools/how-to-guides/data-source-plugins/add-authentication-for-data-source-plugins).",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"description": "",
|
||||
"description": "For data source plugins. Proxy routes used for plugin authentication and adding headers to HTTP requests made by the plugin. For more information, refer to [Authentication for data source plugins](https://grafana.com/developers/plugin-tools/how-to-guides/data-source-plugins/add-authentication-for-data-source-plugins).",
|
||||
"additionalProperties": false,
|
||||
"properties": {
|
||||
"path": {
|
||||
|
||||
+12
-7
@@ -38,13 +38,6 @@ Users can now view anonymous usage statistics, including the count of devices an
|
||||
|
||||
The number of anonymous devices is not limited by default. The configuration option `device_limit` allows you to enforce a limit on the number of anonymous devices. This enables you to have greater control over the usage within your Grafana instance and keep the usage within the limits of your environment. Once the limit is reached, any new devices that try to access Grafana will be denied access.
|
||||
|
||||
To display anonymous users and devices for versions 10.2, 10.3, 10.4, you need to enable the feature toggle `displayAnonymousStats`
|
||||
|
||||
```bash
|
||||
[feature_toggles]
|
||||
enable = displayAnonymousStats
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Example:
|
||||
@@ -67,3 +60,15 @@ device_limit =
|
||||
```
|
||||
|
||||
If you change your organization name in the Grafana UI this setting needs to be updated to match the new name.
|
||||
|
||||
## Licensing for anonymous access
|
||||
|
||||
Grafana Enterprise (self-managed) licenses anonymous access as active users.
|
||||
|
||||
Anonymous access lets people use Grafana without login credentials. It was an early way to share dashboards, but Public dashboards gives you a more secure way to share dashboards.
|
||||
|
||||
### How anonymous usage is counted
|
||||
|
||||
Grafana estimates anonymous active users from anonymous devices:
|
||||
|
||||
- **Counting rule**: Grafana counts 1 anonymous user for every 3 anonymous devices detected.
|
||||
|
||||
@@ -54,18 +54,6 @@ SCIM offers several advantages for managing users and teams in Grafana:
|
||||
|
||||
## Authentication and access requirements
|
||||
|
||||
{{< admonition type="warning" title="Critical: Aligning SAML Identifier with SCIM externalId" >}}
|
||||
When using SAML for authentication alongside SCIM provisioning, a critical security measure is to ensure proper alignment between the the SCIM user's `externalId` and the SAML user identifier. The unique identifier used for SCIM provisioning (which becomes the `externalId` in Grafana, often sourced from a stable IdP attribute like Entra ID's `user.objectid`) **must also be sent as a claim in the SAML assertion from your Identity Provider.**
|
||||
Furthermore, the Grafana SAML configuration must be correctly set up to identify and use this specific claim for linking the authenticated SAML user to their SCIM-provisioned user. This can be achieved by either ensuring the primary SAML login identifier by using the `assertion_attribute_external_uid` setting in Grafana to explicitly set the name of the SAML claim that contains the stable unique identifier attribute.
|
||||
|
||||
**Why is this important?**
|
||||
A mismatch or inconsistent mapping between this SAML login identifier and the SCIM `externalId` creates a critical security vulnerability. If these two identifiers are not reliably and uniquely aligned for each individual user, Grafana may fail to correctly link an authenticated SAML session to the intended SCIM-provisioned user profile and its associated permissions. This can enable a malicious actor to impersonate another user—for instance, by crafting a SAML assertion that, due to the identifier misalignment, incorrectly grants them the access rights of the targeted user.
|
||||
|
||||
Grafana relies on this linkage to correctly associate the authenticated user from SAML with the provisioned user from SCIM. Failure to ensure a consistent and unique identifier across both systems can break this linkage, leading to incorrect user mapping and potential unauthorized access.
|
||||
|
||||
Always verify that your SAML identity provider is configured to send a stable, unique user identifier that your SCIM configuration maps to `externalId`. Refer to your identity provider's documentation and the specific Grafana SCIM integration guides (e.g., for [Entra ID](configure-scim-with-azuread/) or [Okta](configure-scim-with-okta/)) for detailed instructions on configuring these attributes correctly.
|
||||
{{< /admonition >}}
|
||||
|
||||
When you enable SCIM in Grafana, the following requirements and restrictions apply:
|
||||
|
||||
1. **Use the same identity provider for user provisioning and for authentication flow**: You must use the same identity provider for both authentication and user provisioning.
|
||||
@@ -74,6 +62,12 @@ When you enable SCIM in Grafana, the following requirements and restrictions app
|
||||
- Configure `userUID` SAML assertion in [Entra ID](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/configure-saml-with-azuread/#configure-saml-assertions-when-using-scim-provisioning)
|
||||
- Configure `userUID` SAML assertion in [Okta](/docs/grafana/<GRAFANA_VERSION>/setup-grafana/configure-access/configure-authentication/saml/configure-saml-with-okta/#configure-saml-assertions-when-using-scim-provisioning)
|
||||
|
||||
### Align SAML identifier with SCIM `externalId`
|
||||
|
||||
When you use SAML with SCIM provisioning, align the SCIM `externalId` with the SAML user identifier. Use a stable IdP attribute (for example, Entra ID `user.objectid`) as the SCIM `externalId`, and send that same value as a SAML claim. Configure Grafana to read this claim with the `assertion_attribute_external_uid` setting so SAML authentication links to the SCIM-provisioned user and its permissions.
|
||||
|
||||
If the SAML identifier and SCIM `externalId` differ, Grafana may not link the authenticated user to the intended SCIM profile, which can result in incorrect access. Verify your IdP sends a stable, unique identifier and that it matches the SCIM `externalId`. Refer to your IdP docs and the Grafana SCIM integration guides for [Entra ID](configure-scim-with-azuread/) and [Okta](configure-scim-with-okta/) for attribute configuration details.
|
||||
|
||||
## Configure SCIM using the Grafana user interface
|
||||
|
||||
You can configure SCIM in Grafana using the Grafana user interface. To do this, navigate to **Administration > Authentication > SCIM**.
|
||||
|
||||
@@ -642,6 +642,12 @@ You must also provide the `rudderstack_write_key` to enable this feature.
|
||||
Optional.
|
||||
If tracking with RudderStack is enabled, you can provide a custom URL to load the RudderStack SDK.
|
||||
|
||||
#### `rudderstack_v3_sdk_url`
|
||||
|
||||
Optional.
|
||||
This is mirroring the old configuration option, which will be deprecated.
|
||||
If `rudderstack_sdk_url` and `rudderstack_v3_sdk_url` are both set, the feature toggle `rudderstackUpgrade` will control which one is loaded.
|
||||
|
||||
#### `rudderstack_config_url`
|
||||
|
||||
Optional.
|
||||
@@ -2046,6 +2052,10 @@ This section applies only to rules imported as Grafana-managed rules. For more i
|
||||
|
||||
Set the query offset to imported Grafana-managed rules when `query_offset` is not defined in the original rule group configuration. The default value is `1m`.
|
||||
|
||||
#### `default_datasource_uid`
|
||||
|
||||
Set the default data source UID to use for query execution when importing Prometheus rules. Grafana uses this default when the `X-Grafana-Alerting-Datasource-UID` header isn't provided during import. If this option isn't set, the header becomes required. The default value is empty.
|
||||
|
||||
<hr>
|
||||
|
||||
### `[annotations]`
|
||||
|
||||
@@ -31,7 +31,6 @@ Most [generally available](https://grafana.com/docs/release-life-cycle/#general-
|
||||
| `logsContextDatasourceUi` | Allow datasource to provide custom UI for context view | Yes |
|
||||
| `lokiQuerySplitting` | Split large interval queries into subqueries with smaller time intervals | Yes |
|
||||
| `influxdbBackendMigration` | Query InfluxDB InfluxQL without the proxy | Yes |
|
||||
| `unifiedRequestLog` | Writes error logs to the request logger | Yes |
|
||||
| `logsExploreTableVisualisation` | A table visualisation for logs in Explore | Yes |
|
||||
| `awsDatasourcesTempCredentials` | Support temporary security credentials in AWS plugins for Grafana Cloud customers | Yes |
|
||||
| `awsAsyncQueryCaching` | Enable caching for async queries for Redshift and Athena. Requires that the datasource has caching and async query support enabled | Yes |
|
||||
|
||||
@@ -46,7 +46,7 @@ Complete the following steps to install Grafana from the APT repository:
|
||||
1. Install the prerequisite packages:
|
||||
|
||||
```bash
|
||||
sudo apt-get install -y apt-transport-https software-properties-common wget
|
||||
sudo apt-get install -y apt-transport-https wget
|
||||
```
|
||||
|
||||
1. Import the GPG key:
|
||||
|
||||
@@ -135,9 +135,12 @@ You can use the **Span Limit** field in **Options** section of the TraceQL query
|
||||
This field sets the maximum number of spans to return for each span set.
|
||||
By default, the maximum value that you can set for the **Span Limit** value (or the spss query) is 100.
|
||||
In Tempo configuration, this value is controlled by the `max_spans_per_span_set` parameter and can be modified by your Tempo administrator.
|
||||
Grafana Cloud users can contact Grafana Support to request a change.
|
||||
Entering a value higher than the default results in an error.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Changing the value of `max_spans_per_span_set` isn't supported in Grafana Cloud.
|
||||
{{< /admonition >}}
|
||||
|
||||
### Focus on traces or spans
|
||||
|
||||
Under **Options**, you can choose to display the table as **Traces** or **Spans** focused.
|
||||
|
||||
@@ -23,6 +23,8 @@ killercoda:
|
||||
|
||||
This tutorial is a continuation of the [Get started with Grafana Alerting - Route alerts using dynamic labels](http://www.grafana.com/tutorials/alerting-get-started-pt5/) tutorial.
|
||||
|
||||
{{< youtube id="mqj_hN24zLU" >}}
|
||||
|
||||
<!-- USE CASE -->
|
||||
|
||||
In this tutorial you will learn how to:
|
||||
|
||||
+2
-2
@@ -163,9 +163,9 @@ To add a new annotation query to a dashboard, follow these steps:
|
||||
|
||||
1. To create a query, do one of the following:
|
||||
- Write or construct a query in the query language of your data source. The annotation query options are different for each data source. For information about annotations in a specific data source, refer to the specific [data source](ref:data-source) topic.
|
||||
- Click **Replace with saved query** to reuse a [saved query](ref:saved-queries).
|
||||
- Open the **Saved queries** drop-down menu and click **Replace query** to reuse a [saved query](ref:saved-queries).
|
||||
|
||||
1. (Optional) To [save the query](ref:save-query) for reuse, click the **Save query** button (or icon).
|
||||
1. (Optional) To [save the query](ref:save-query) for reuse, open the **Saved queries** drop-down menu and click the **Save query** option.
|
||||
1. (Optional) Click **Test annotation query** to ensure that the query is working properly.
|
||||
1. (Optional) To add subsequent queries, click **+ Add query** or **+ Add from saved queries**, and test them as many times as needed.
|
||||
|
||||
|
||||
@@ -27,7 +27,7 @@ Dashboards and panels allow you to show your data in visual form. Each panel nee
|
||||
- Understand the query language of the target data source.
|
||||
- Ensure that data source for which you are writing a query has been added. For more information about adding a data source, refer to [Add a data source](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/datasources/#add-a-data-source) if you need instructions.
|
||||
|
||||
**To create a dashboard**:
|
||||
To create a dashboard, follow these steps:
|
||||
|
||||
{{< docs/list >}}
|
||||
{{< shared id="create-dashboard" >}}
|
||||
@@ -54,9 +54,9 @@ Dashboards and panels allow you to show your data in visual form. Each panel nee
|
||||
|
||||
1. To create a query, do one of the following:
|
||||
- Write or construct a query in the query language of your data source.
|
||||
- Click **Replace with saved query** to reuse a [saved query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries).
|
||||
- Open the **Saved queries** drop-down menu and click **Replace query** to reuse a [saved query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#saved-queries).
|
||||
|
||||
1. (Optional) To [save the query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#save-a-query) for reuse, click the **Save query** button (or icon).
|
||||
1. (Optional) To [save the query](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/query-transform-data/#save-a-query) for reuse, open the **Saved queries** drop-down menu and click the **Save query** option.
|
||||
1. Click **Refresh** to query the data source.
|
||||
1. (Optional) To add subsequent queries, click **+ Add query** or **+ Add from saved queries**, and refresh the data source as many times as needed.
|
||||
|
||||
@@ -102,6 +102,28 @@ Dashboards and panels allow you to show your data in visual form. Each panel nee
|
||||
|
||||
{{< /docs/list >}}
|
||||
|
||||
### Begin dashboard creation from data source configuration
|
||||
|
||||
You can start the process of creating a dashboard directly from a data source rather than from the **Dashboards** page.
|
||||
|
||||
To begin building a dashboard directly from a data source, follow these steps:
|
||||
|
||||
1. Navigate to **Connections > Data sources**.
|
||||
1. On the row of the data source for which you want to build a dashboard, click **Build a dashboard**.
|
||||
|
||||
The empty dashboard page opens.
|
||||
|
||||
1. Do one of the following:
|
||||
- Click **+Add visualization** to configure all the elements of the new dashboard.
|
||||
- Select one of the suggested dashboards by clicking its **Use dashboard** button. This can be helpful when you're not sure how to most effectively visualize your data.
|
||||
The suggested dashboards are specific to your data source type (for example, Prometheus, Loki, or Elasticsearch). If there are more than three dashboard suggestions, you can click **View all** to see the rest of them.
|
||||
|
||||
|
||||
|
||||
{{< docs/public-preview product="Suggested dashboards" >}}
|
||||
|
||||
1. Complete the rest of the dashboard configuration. For more detailed steps, refer to [Create a dashboard](#create-a-dashboard), beginning at step five.
|
||||
|
||||
## Copy a dashboard
|
||||
|
||||
To copy a dashboard, follow these steps:
|
||||
|
||||
@@ -124,6 +124,8 @@ For more information about dashboard permissions, refer to [Dashboard permission
|
||||
## Restore deleted dashboards
|
||||
|
||||
{{% admonition type="caution" %}}
|
||||
Restoring deleted dashboards is currently in private preview. Grafana Labs offers support on a best-effort basis, and breaking changes might occur prior to the feature being made generally available.
|
||||
|
||||
The feature is only available in Grafana Cloud.
|
||||
{{% /admonition %}}
|
||||
|
||||
|
||||
@@ -98,7 +98,7 @@ You can share dashboards in the following ways:
|
||||
- [As a report](#schedule-a-report)
|
||||
- [As a snapshot](#share-a-snapshot)
|
||||
- [As a PDF export](#export-a-dashboard-as-pdf)
|
||||
- [As a JSON file export](#export-a-dashboard-as-json)
|
||||
- [As a JSON file export](#export-a-dashboard-as-code)
|
||||
- [As an image export](#export-a-dashboard-as-an-image)
|
||||
|
||||
When you share a dashboard externally as a link or by email, those dashboards are included in a list of your shared dashboards. To view the list and manage these dashboards, navigate to **Dashboards > Shared dashboards**.
|
||||
|
||||
@@ -71,8 +71,9 @@ Explore consists of a toolbar, outline, query editor, the ability to add multipl
|
||||
- **Run query** - Click to run your query.
|
||||
|
||||
- **Query editor** - Interface where you construct the query for a specific data source. Query editor elements differ based on data source. In order to run queries across multiple data sources you need to select **Mixed** from the data source picker.
|
||||
- **Save query** - To [save the query](ref:save-query) for reuse, click the **Save query** button (or icon).
|
||||
- **Replace with saved query** - Reuse a saved query.
|
||||
- **Saved queries**:
|
||||
- **Save query** - To [save the query](ref:save-query) for reuse, click the **Save query** button (or icon).
|
||||
- **Replace query** - Reuse a saved query.
|
||||
- **+ Add query** - Add an additional query.
|
||||
- **+ Add from saved queries** - Add an additional query by reusing a saved query.
|
||||
|
||||
|
||||
@@ -88,8 +88,9 @@ The data section contains tabs where you enter queries, transform your data, and
|
||||
|
||||
- **Queries**
|
||||
- Select your data source. You can also set or update the data source in existing dashboards using the drop-down menu in the **Queries** tab.
|
||||
- **Save query** - To [save the query](ref:save-query) for reuse, click the **Save query** button (or icon).
|
||||
- **Replace with saved query** - Reuse a saved query.
|
||||
- **Saved queries**:
|
||||
- **Save query** - To [save the query](ref:save-query) for reuse, click the **Save query** button (or icon).
|
||||
- **Replace query** - Reuse a saved query.
|
||||
- **+ Add query** - Add an additional query.
|
||||
- **+ Add from saved queries** - Add an additional query by reusing a saved query.
|
||||
|
||||
|
||||
@@ -156,11 +156,11 @@ In the **Saved queries** drawer, you can:
|
||||
- Edit a query title, description, tags, or the availability of the query to other users in your organization. By default, saved queries are locked for editing.
|
||||
- When you access the **Saved queries** drawer from Explore, you can use the **Edit in Explore** option to edit the body of a query.
|
||||
|
||||
To access your saved queries, click **+ Add from saved queries** or **Replace with saved query** in the query editor:
|
||||
To access your saved queries, click **+ Add from saved queries** or open the **Saved queries** drop-down menu and click **Replace query** in the query editor:
|
||||
|
||||
{{< figure src="/media/docs/grafana/dashboards/screenshot-use-saved-queries-v12.3.png" max-width="750px" alt="Access saved queries" >}}
|
||||
|
||||
Clicking **+ Add from saved queries** adds an additional query, while clicking **Replace with saved query** updates your existing query.
|
||||
Clicking **+ Add from saved queries** adds an additional query, while clicking **Replace query** in the **Saved queries** drop-down menu updates your existing query.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Users with Admin and Editor roles can create and save queries for reuse.
|
||||
@@ -172,7 +172,7 @@ Viewers can only reuse queries.
|
||||
|
||||
To save a query you've created:
|
||||
|
||||
1. From the query editor, click the **Save query** icon:
|
||||
1. From the query editor, open the **Saved queries** drop-down menu and click the **Save query** option:
|
||||
|
||||
{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-save-query-v12.2.png" max-width="750px" alt="Save a query" >}}
|
||||
|
||||
@@ -227,7 +227,7 @@ To add a query, follow these steps:
|
||||
|
||||
1. To create a query, do one of the following:
|
||||
- Write or construct a query in the query language of your data source.
|
||||
- Click **Replace with saved query** to reuse a saved query.
|
||||
- Open the **Saved queries** drop-down menu and click **Replace query** to reuse a saved query.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
[Saved queries](#saved-queries) is currently in [public preview](https://grafana.com/docs/release-life-cycle/). Grafana Labs offers limited support, and breaking changes might occur prior to the feature being made generally available.
|
||||
@@ -235,7 +235,7 @@ To add a query, follow these steps:
|
||||
This feature is only available on Grafana Enterprise and Grafana Cloud.
|
||||
{{< /admonition >}}
|
||||
|
||||
1. (Optional) To [save the query](#save-a-query) for reuse, click the **Save query** button (or icon).
|
||||
1. (Optional) To [save the query](#save-a-query) for reuse, click the **Save query** option in the **Saved queries** drop-down menu.
|
||||
1. (Optional) Click **+ Add query** or **Add from saved queries** to add more queries as needed.
|
||||
|
||||
1. Click **Run queries**.
|
||||
|
||||
Reference in New Issue
Block a user