Compare commits
13 Commits
feature/ex
...
docs/loki-
| Author | SHA1 | Date | |
|---|---|---|---|
|
0f1de558e2 | ||
|
|
ba2ef712d8 | ||
|
|
a45400de8d | ||
|
|
0c7936111f | ||
|
|
7c55021b94 | ||
|
|
8b7bd6f646 | ||
|
|
b889d23c29 | ||
|
|
e014d9a000 | ||
|
|
3eab41557a | ||
|
|
db96b6c1e3 | ||
|
|
3f78facce0 | ||
|
|
361fadf3be | ||
|
|
f244c9d483 |
@@ -14,147 +14,140 @@ labels:
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Loki
|
||||
title: Configure the Loki data source
|
||||
title: Loki data source
|
||||
weight: 800
|
||||
refs:
|
||||
data-source-management:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
build-dashboards:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/
|
||||
provisioning-data-sources:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
explore:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
logs-integration-labels-and-detected-fields:
|
||||
visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/visualizations/
|
||||
variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/dashboards/variables/
|
||||
transformations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/query-transform-data/transform-data/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/panels-visualizations/query-transform-data/transform-data/
|
||||
loki-alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/alerting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/alerting/
|
||||
loki-annotations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/annotations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/annotations/
|
||||
import-dashboard:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/import-dashboards/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/visualizations/dashboards/build-dashboards/import-dashboards/
|
||||
loki-troubleshooting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
loki-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/
|
||||
loki-template-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
configure-loki-derived-fields:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/#derived-fields
|
||||
---
|
||||
|
||||
# Loki data source
|
||||
|
||||
Grafana Loki is a set of components that can be combined into a fully featured logging stack.
|
||||
Unlike other logging systems, Loki is built around the idea of only indexing metadata about your logs: labels (just like Prometheus labels). Log data itself is then compressed and stored in chunks in object stores such as S3 or GCS, or even locally on a filesystem.
|
||||
Grafana Loki is a log aggregation system that stores and queries logs from your applications and infrastructure. Unlike traditional logging systems, Loki indexes only metadata (labels) about your logs rather than the full text. Log data is compressed and stored in object stores such as Amazon S3 or Google Cloud Storage, or locally on a filesystem.
|
||||
|
||||
The following guides will help you get started with Loki:
|
||||
|
||||
- [Getting started with Loki](/docs/loki/latest/get-started/)
|
||||
- [Install Loki](/docs/loki/latest/installation/)
|
||||
- [Loki best practices](/docs/loki/latest/best-practices/#best-practices)
|
||||
- [Configure the Loki data source](/docs/grafana/latest/datasources/loki/configure-loki-data-source/)
|
||||
- [LogQL](/docs/loki/latest/logql/)
|
||||
- [Loki query editor](query-editor/)
|
||||
You can use this data source to query, visualize, and alert on log data stored in Loki.
|
||||
|
||||
## Supported Loki versions
|
||||
|
||||
This data source supports these versions of Loki:
|
||||
This data source supports Loki v2.9 and later.
|
||||
|
||||
- v2.9+
|
||||
## Key capabilities
|
||||
|
||||
## Adding a data source
|
||||
The Loki data source provides the following capabilities:
|
||||
|
||||
For instructions on how to add a data source to Grafana, refer to the [administration documentation](ref:data-source-management)
|
||||
Only users with the organization administrator role can add data sources.
|
||||
Administrators can also [configure the data source via YAML](#provision-the-data-source) with Grafana's provisioning system.
|
||||
- **Log queries:** Query and filter logs using [LogQL](https://grafana.com/docs/loki/latest/logql/), Loki's query language inspired by PromQL.
|
||||
- **Metric queries:** Extract metrics from log data using LogQL metric queries, enabling you to count log events, calculate rates, and aggregate values.
|
||||
- **Live tailing:** Stream logs in real time as they're ingested into Loki.
|
||||
- **Derived fields:** Create links from log lines to external systems such as tracing backends, allowing you to jump directly from a log entry to a related trace.
|
||||
- **Annotations:** Overlay log events on time series graphs to correlate logs with metrics.
|
||||
- **Alerting:** Create alert rules based on log queries to notify you when specific patterns or thresholds are detected.
|
||||
|
||||
Once you've added the Loki data source, you can [configure it](#configure-the-data-source) so that your Grafana instance's users can create queries in its [query editor](query-editor/) when they [build dashboards](ref:build-dashboards), use [Explore](ref:explore), and [annotate visualizations](query-editor/#apply-annotations).
|
||||
## Get started
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
|
||||
{{< /admonition >}}
|
||||
The following documentation helps you get started with the Loki data source:
|
||||
|
||||
## Provision the data source
|
||||
- [Configure the Loki data source](ref:configure-loki)
|
||||
- [Loki query editor](ref:loki-query-editor)
|
||||
- [Loki template variables](ref:loki-template-variables)
|
||||
- [Troubleshoot the Loki data source](ref:loki-troubleshooting)
|
||||
|
||||
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
|
||||
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources).
|
||||
For more information about Loki itself, refer to the [Loki documentation](https://grafana.com/docs/loki/latest/):
|
||||
|
||||
### Provisioning examples
|
||||
- [Get started with Loki](https://grafana.com/docs/loki/latest/get-started/)
|
||||
- [Install Loki](https://grafana.com/docs/loki/latest/installation/)
|
||||
- [Loki best practices](https://grafana.com/docs/loki/latest/best-practices/#best-practices)
|
||||
- [LogQL query language](https://grafana.com/docs/loki/latest/logql/)
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
## Additional features
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://localhost:3100
|
||||
jsonData:
|
||||
timeout: 60
|
||||
maxLines: 1000
|
||||
```
|
||||
After you configure the Loki data source, you can:
|
||||
|
||||
**Using basic authorization and a derived field:**
|
||||
- Create [visualizations](ref:visualizations) to display your log data
|
||||
- Configure and use [templates and variables](ref:variables) for dynamic dashboards
|
||||
- Add [transformations](ref:transformations) to process query results
|
||||
- Add [annotations](ref:loki-annotations) to overlay log events on graphs
|
||||
- Set up [alerting](ref:loki-alerting) to monitor your log data
|
||||
- Use [Explore](ref:explore) for ad-hoc log queries and analysis
|
||||
- Configure [derived fields](ref:configure-loki-derived-fields) to link logs to traces or other data sources
|
||||
|
||||
You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables:
|
||||
If you encounter issues, refer to [Troubleshoot issues with the Loki data source](ref:loki-troubleshooting).
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
## Community dashboards
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://localhost:3100
|
||||
basicAuth: true
|
||||
basicAuthUser: my_user
|
||||
jsonData:
|
||||
maxLines: 1000
|
||||
derivedFields:
|
||||
# Field with internal link pointing to data source in Grafana.
|
||||
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
|
||||
- datasourceUid: my_jaeger_uid
|
||||
matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
# url will be interpreted as query for the datasource
|
||||
url: '$${__value.raw}'
|
||||
# optional for URL Label to set a custom display label for the link.
|
||||
urlDisplayLabel: 'View Trace'
|
||||
Grafana doesn't ship pre-configured dashboards with the Loki data source, but you can find community-contributed dashboards on [Grafana Dashboards](https://grafana.com/grafana/dashboards/?dataSource=loki). These dashboards provide ready-made visualizations for common Loki use cases.
|
||||
|
||||
# Field with external link.
|
||||
- matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
url: 'http://localhost:16686/trace/$${__value.raw}'
|
||||
secureJsonData:
|
||||
basicAuthPassword: test_password
|
||||
```
|
||||
To import a community dashboard:
|
||||
|
||||
**Using a Jaeger data source:**
|
||||
1. Find a dashboard on [grafana.com/grafana/dashboards](https://grafana.com/grafana/dashboards/?dataSource=loki).
|
||||
1. Copy the dashboard ID.
|
||||
1. In Grafana, go to **Dashboards** > **New** > **Import**.
|
||||
1. Paste the dashboard ID and click **Load**.
|
||||
|
||||
In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value.
|
||||
For more information, refer to [Import a dashboard](ref:import-dashboard).
|
||||
|
||||
```
|
||||
datasources:
|
||||
- name: Jaeger
|
||||
type: jaeger
|
||||
url: http://jaeger-tracing-query:16686/
|
||||
access: proxy
|
||||
# UID should match the datasourceUid in derivedFields.
|
||||
uid: my_jaeger_uid
|
||||
```
|
||||
## Related data sources
|
||||
|
||||
## Query the data source
|
||||
Loki integrates with other Grafana data sources to provide full observability across logs, metrics, and traces:
|
||||
|
||||
The Loki data source's query editor helps you create log and metric queries that use Loki's query language, [LogQL](/docs/loki/latest/logql/).
|
||||
- **Tempo:** Use [derived fields](ref:configure-loki-derived-fields) to create links from log lines to traces in Tempo, enabling seamless navigation from logs to distributed traces.
|
||||
- **Prometheus and Mimir:** Display logs alongside metrics on the same dashboard to correlate application behavior with performance data.
|
||||
|
||||
For details, refer to the [query editor documentation](query-editor/).
|
||||
|
||||
## Use template variables
|
||||
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
|
||||
Grafana refers to such variables as template variables.
|
||||
|
||||
For details, see the [template variables documentation](template-variables/).
|
||||
For more information about building observability workflows, refer to the [Grafana Tempo documentation](https://grafana.com/docs/tempo/latest/) and [Grafana Mimir documentation](https://grafana.com/docs/mimir/latest/).
|
||||
|
||||
226
docs/sources/datasources/loki/alerting/index.md
Normal file
226
docs/sources/datasources/loki/alerting/index.md
Normal file
@@ -0,0 +1,226 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/loki/alerting/
|
||||
description: Use Grafana Alerting with the Loki data source
|
||||
keywords:
|
||||
- grafana
|
||||
- loki
|
||||
- alerting
|
||||
- alerts
|
||||
- logs
|
||||
- recording rules
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Alerting
|
||||
title: Loki alerting
|
||||
weight: 450
|
||||
refs:
|
||||
alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/
|
||||
create-alert-rule:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-grafana-managed-rule/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-grafana-managed-rule/
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/
|
||||
grafana-managed-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/
|
||||
data-source-managed-recording-rules:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/alerting-rules/create-recording-rules/create-data-source-managed-recording-rules/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/alerting-rules/create-recording-rules/create-data-source-managed-recording-rules/
|
||||
---
|
||||
|
||||
# Loki alerting
|
||||
|
||||
You can use Grafana Alerting with Loki to create alerts based on your log data. This allows you to monitor error rates, detect patterns, and receive notifications when specific conditions are met in your logs.
|
||||
|
||||
For general information about Grafana Alerting, refer to [Grafana Alerting](ref:alerting).
|
||||
|
||||
## Before you begin
|
||||
|
||||
Before creating alerts with Loki, ensure you have:
|
||||
|
||||
- A [Loki data source configured](ref:configure-loki) in Grafana.
|
||||
- Appropriate permissions to create alert rules.
|
||||
- Understanding of the log patterns you want to monitor.
|
||||
- The **Manage alert rules in Alerting UI** toggle enabled in the Loki data source settings.
|
||||
|
||||
## Supported query types
|
||||
|
||||
Loki alerting requires **metric queries** that return numeric time series data. You must use LogQL metric queries that wrap log stream selectors with aggregation functions.
|
||||
|
||||
### Query types and alerting compatibility
|
||||
|
||||
| Query type | Alerting support | Notes |
|
||||
| ------------- | ---------------- | ----------------------------------------------- |
|
||||
| Metric query | ✅ Full support | Use range aggregation functions like `rate()` |
|
||||
| Log query | ❌ Not supported | Convert to metric query using aggregations |
|
||||
| Instant query | ⚠️ Limited | Range queries recommended for time-based alerts |
|
||||
|
||||
### Common metric functions for alerting
|
||||
|
||||
Use these LogQL functions to convert log queries into metric queries suitable for alerting:
|
||||
|
||||
| Function | Description | Example |
|
||||
| -------------------- | ---------------------------------------------- | --------------------------------------------------- |
|
||||
| `rate()` | Rate of log entries per second | `rate({job="app"}[5m])` |
|
||||
| `count_over_time()` | Count of log entries in the specified interval | `count_over_time({job="app"}[5m])` |
|
||||
| `sum_over_time()` | Sum of extracted numeric values | `sum_over_time({job="app"} \| unwrap latency [5m])` |
|
||||
| `avg_over_time()` | Average of extracted numeric values | `avg_over_time({job="app"} \| unwrap latency [5m])` |
|
||||
| `max_over_time()` | Maximum extracted value in the interval | `max_over_time({job="app"} \| unwrap latency [5m])` |
|
||||
| `bytes_rate()` | Rate of bytes per second | `bytes_rate({job="app"}[5m])` |
|
||||
| `absent_over_time()` | Returns 1 if no logs exist in the interval | `absent_over_time({job="app"}[5m])` |
|
||||
|
||||
## Create an alert rule
|
||||
|
||||
To create an alert rule using Loki:
|
||||
|
||||
1. Navigate to **Alerting** > **Alert rules**.
|
||||
1. Click **New alert rule**.
|
||||
1. Enter a name for the alert rule.
|
||||
1. Select your **Loki** data source.
|
||||
1. Build your metric query:
|
||||
- Start with a log stream selector (for example, `{job="app"}`)
|
||||
- Add filters if needed (for example, `|= "error"`)
|
||||
- Wrap with a metric function (for example, `rate(...[5m])`)
|
||||
1. Configure the alert condition (for example, when the rate is above a threshold).
|
||||
1. Set the evaluation interval and pending period.
|
||||
1. Configure notifications and labels.
|
||||
1. Click **Save rule**.
|
||||
|
||||
For detailed instructions, refer to [Create a Grafana-managed alert rule](ref:create-alert-rule).
|
||||
|
||||
## Example alert queries
|
||||
|
||||
The following examples show common alerting scenarios with Loki.
|
||||
|
||||
### Alert on high error rate
|
||||
|
||||
Monitor the rate of error logs:
|
||||
|
||||
```logql
|
||||
rate({job="app"} |= "error" [5m]) > 0.1
|
||||
```
|
||||
|
||||
This query calculates the rate of log lines containing "error" per second over the last 5 minutes and alerts when it exceeds 0.1 errors per second.
|
||||
|
||||
### Alert on error count threshold
|
||||
|
||||
Monitor the count of errors in a time window:
|
||||
|
||||
```logql
|
||||
sum(count_over_time({job="app", level="error"}[15m])) > 100
|
||||
```
|
||||
|
||||
This query counts error-level logs over 15 minutes and alerts when the count exceeds 100.
|
||||
|
||||
### Alert on high latency
|
||||
|
||||
Monitor request latency extracted from logs:
|
||||
|
||||
```logql
|
||||
avg_over_time({job="api"} | logfmt | unwrap duration [5m]) > 500
|
||||
```
|
||||
|
||||
This query extracts the `duration` field from logfmt-formatted logs and alerts when the average exceeds 500 milliseconds.
|
||||
|
||||
### Alert on missing logs
|
||||
|
||||
Detect when a service stops sending logs:
|
||||
|
||||
```logql
|
||||
absent_over_time({job="critical-service"}[10m])
|
||||
```
|
||||
|
||||
This query alerts when no logs are received from the critical service for 10 minutes.
|
||||
|
||||
### Alert by label grouping
|
||||
|
||||
Monitor errors grouped by service:
|
||||
|
||||
```logql
|
||||
sum by (service) (rate({namespace="production"} |= "error" [5m])) > 0.05
|
||||
```
|
||||
|
||||
This query calculates error rates per service and alerts when any service exceeds the threshold.
|
||||
|
||||
## Recording rules
|
||||
|
||||
Recording rules pre-compute frequently used or expensive LogQL queries and save the results as new time series metrics. This improves query performance and reduces load on your Loki instance.
|
||||
|
||||
For detailed information about recording rules, refer to [Create recording rules](ref:recording-rules).
|
||||
|
||||
### Use cases for Loki recording rules
|
||||
|
||||
Recording rules are useful when you need to:
|
||||
|
||||
- **Pre-aggregate expensive queries:** Convert complex log aggregations into simple metric queries.
|
||||
- **Track trends over time:** Create metrics from log data that would otherwise be too expensive to query repeatedly.
|
||||
- **Reuse queries across dashboards:** Compute a metric once and reference it in multiple dashboards and alerts.
|
||||
- **Reduce query latency:** Query precomputed results instead of scanning logs in real time.
|
||||
|
||||
### Types of recording rules
|
||||
|
||||
Loki supports two types of recording rules:
|
||||
|
||||
- **Grafana-managed recording rules:** Query Loki using LogQL and store results in a Prometheus-compatible data source. This is the recommended option. Refer to [Create Grafana-managed recording rules](ref:grafana-managed-recording-rules).
|
||||
- **Data source-managed recording rules:** Define recording rules directly in Loki using the Loki ruler. Refer to [Create data source-managed recording rules](ref:data-source-managed-recording-rules).
|
||||
|
||||
### Example recording rule
|
||||
|
||||
The following example creates a metric that tracks the error rate per service:
|
||||
|
||||
```logql
|
||||
sum by (service) (rate({namespace="production"} |= "error" [5m]))
|
||||
```
|
||||
|
||||
This query runs on a schedule (for example, every minute) and stores the result as a new metric. You can then query this precomputed metric in dashboards and alert rules instead of running the full LogQL query each time.
|
||||
|
||||
## Limitations
|
||||
|
||||
When using Loki with Grafana Alerting, be aware of the following limitations:
|
||||
|
||||
### Template variables not supported
|
||||
|
||||
Alert queries cannot contain template variables. Grafana evaluates alert rules on the backend without dashboard context, so variables like `$job` or `$namespace` are not resolved.
|
||||
|
||||
If your dashboard query uses template variables, create a separate query for alerting with hard-coded values.
|
||||
|
||||
### Log queries not supported
|
||||
|
||||
Queries that return log lines cannot be used for alerting. You must convert log queries to metric queries using aggregation functions like `rate()` or `count_over_time()`.
|
||||
|
||||
### Query time range
|
||||
|
||||
Alert queries use the evaluation interval to determine the time range, not the dashboard time picker. Ensure your metric function intervals (for example, `[5m]`) align with your alert evaluation frequency.
|
||||
|
||||
## Best practices
|
||||
|
||||
Follow these best practices when creating Loki alerts:
|
||||
|
||||
- **Use metric queries:** Always wrap log stream selectors with metric functions for alerting.
|
||||
- **Match intervals:** Align the LogQL time interval (for example, `[5m]`) with your alert evaluation interval.
|
||||
- **Be specific with selectors:** Use precise label selectors to reduce the amount of data scanned.
|
||||
- **Test queries first:** Verify your query returns expected numeric results in Explore before creating an alert.
|
||||
- **Use meaningful thresholds:** Base alert thresholds on historical patterns in your log data.
|
||||
- **Add context with labels:** Include relevant labels in your alert to help with triage.
|
||||
143
docs/sources/datasources/loki/annotations/index.md
Normal file
143
docs/sources/datasources/loki/annotations/index.md
Normal file
@@ -0,0 +1,143 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/loki/annotations/
|
||||
description: Use Loki log events as annotations in Grafana dashboards
|
||||
keywords:
|
||||
- grafana
|
||||
- loki
|
||||
- annotations
|
||||
- events
|
||||
- logs
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Annotations
|
||||
title: Loki annotations
|
||||
weight: 400
|
||||
refs:
|
||||
annotate-visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
---
|
||||
|
||||
# Loki annotations
|
||||
|
||||
Annotations overlay event data on your dashboard graphs, helping you correlate log events with metrics. You can use Loki as a data source for annotations to display events such as deployments, errors, or other significant occurrences on your visualizations.
|
||||
|
||||
For general information about annotations, refer to [Annotate visualizations](ref:annotate-visualizations).
|
||||
|
||||
## Before you begin
|
||||
|
||||
Before creating Loki annotations, ensure you have:
|
||||
|
||||
- A [Loki data source configured](ref:configure-loki) in Grafana.
|
||||
- Logs in Loki containing the events you want to display as annotations.
|
||||
- Read access to the Loki logs you want to query.
|
||||
|
||||
## Create an annotation query
|
||||
|
||||
To add a Loki annotation to your dashboard:
|
||||
|
||||
1. Navigate to your dashboard and click **Dashboard settings** (gear icon).
|
||||
1. Select **Annotations** in the left menu.
|
||||
1. Click **Add annotation query**.
|
||||
1. Enter a **Name** for the annotation.
|
||||
1. Select your **Loki** data source from the **Data source** dropdown.
|
||||
1. Enter a LogQL query in the query field.
|
||||
1. Configure the optional formatting fields (Title, Tags, Text).
|
||||
1. Click **Save dashboard**.
|
||||
|
||||
## Query
|
||||
|
||||
Use the query field to enter a LogQL expression that filters the log events to display as annotations. Only log queries are supported for annotations; metric queries are not supported.
|
||||
|
||||
**Examples:**
|
||||
|
||||
| Query | Description |
|
||||
| ----------------------------------------- | ------------------------------------------------- |
|
||||
| `{job="app"}` | Shows all logs from the "app" job. |
|
||||
| `{job="app"} \|= "error"` | Shows logs containing "error" from the "app" job. |
|
||||
| `{namespace="production"} \|= "deployed"` | Shows deployment events in production. |
|
||||
| `{job="app"} \| logfmt \| level="error"` | Shows error-level logs using logfmt parsing. |
|
||||
| `{job="$job"}` | Uses a template variable to filter by job. |
|
||||
|
||||
You can use template variables in your annotation queries to make them dynamic based on dashboard selections.
|
||||
|
||||
## Formatting options
|
||||
|
||||
Loki annotations support optional formatting fields to customize how annotations are displayed.
|
||||
|
||||
### Title
|
||||
|
||||
The **Title** field specifies a pattern for the annotation title. You can use label values by wrapping the label name in double curly braces.
|
||||
|
||||
- **Default:** Empty (uses the log line as the title)
|
||||
- **Pattern example:** `{{instance}}` displays the value of the `instance` label
|
||||
- **Pattern example:** `{{job}} - {{level}}` combines multiple labels
|
||||
|
||||
### Tags
|
||||
|
||||
The **Tags** field specifies which labels to use as annotation tags. Enter label names as a comma-separated list.
|
||||
|
||||
- **Default:** All labels are used as tags
|
||||
- **Example:** `job,instance,level` uses only these three labels as tags
|
||||
|
||||
Tags help categorize and filter annotations in the dashboard.
|
||||
|
||||
### Text
|
||||
|
||||
The **Text** field specifies a pattern for the annotation text displayed when you hover over the annotation. You can use label values by wrapping the label name in double curly braces.
|
||||
|
||||
- **Default:** The log line content
|
||||
- **Pattern example:** `{{message}}` displays the value of a parsed `message` label
|
||||
- **Pattern example:** `Error on {{instance}}: {{error}}` creates a descriptive message
|
||||
|
||||
### Line limit
|
||||
|
||||
The **Line limit** field controls the maximum number of log lines returned for annotations. This helps prevent performance issues when querying logs with many results.
|
||||
|
||||
- **Default:** Uses the data source's configured maximum lines setting
|
||||
|
||||
## Example: Deployment annotations
|
||||
|
||||
To display deployment events as annotations:
|
||||
|
||||
1. Create an annotation query with the following settings:
|
||||
- **Query:** `{job="deploy-service"} |= "deployed"`
|
||||
- **Title:** `Deployment: {{app}}`
|
||||
- **Tags:** `app,environment`
|
||||
- **Text:** `{{message}}`
|
||||
|
||||
This configuration displays deployment logs with the application name in the title and environment as a tag.
|
||||
|
||||
## Example: Error annotations
|
||||
|
||||
To overlay error events on your metrics graphs:
|
||||
|
||||
1. Create an annotation query with the following settings:
|
||||
- **Query:** `{namespace="production"} | logfmt | level="error"`
|
||||
- **Title:** `{{job}} error`
|
||||
- **Tags:** `job,instance`
|
||||
|
||||
This configuration displays error logs from production, grouped by job and instance.
|
||||
|
||||
## Example: Filter annotations with template variables
|
||||
|
||||
To create dynamic annotations that respond to dashboard variable selections:
|
||||
|
||||
1. Create a template variable named `job` that queries Loki label values.
|
||||
1. Create an annotation query with the following settings:
|
||||
- **Query:** `{job="$job"} |= "alert"`
|
||||
- **Title:** `Alert: {{alertname}}`
|
||||
- **Tags:** `severity`
|
||||
|
||||
This configuration displays only alerts for the selected job, making the annotations relevant to the current dashboard context.
|
||||
@@ -1,146 +0,0 @@
|
||||
---
|
||||
aliases:
|
||||
- ../data-sources/loki/
|
||||
- ../features/datasources/loki/
|
||||
description: Configure the Loki data source
|
||||
keywords:
|
||||
- grafana
|
||||
- loki
|
||||
- logging
|
||||
- guide
|
||||
- data source
|
||||
menuTitle: Configure Loki
|
||||
title: Configure the Loki data source
|
||||
weight: 200
|
||||
refs:
|
||||
log-details:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
---
|
||||
|
||||
# Loki data source
|
||||
|
||||
Grafana ships with built-in support for [Loki](/docs/loki/latest/), an open-source log aggregation system by Grafana Labs. If you are new to Loki the following documentation will help you get started:
|
||||
|
||||
- [Getting started](/docs/loki/latest/get-started/)
|
||||
- [Best practices](/docs/loki/latest/best-practices/#best-practices)
|
||||
|
||||
## Configure the Loki data source
|
||||
|
||||
To add the Loki data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under **Connections**, click **Add new connection**.
|
||||
1. Enter `Loki` in the search bar.
|
||||
1. Select **Loki data source**.
|
||||
1. Click **Create a Loki data source** in the upper right.
|
||||
|
||||
You will be taken to the **Settings** tab where you will set up your Loki configuration.
|
||||
|
||||
## Configuration options
|
||||
|
||||
The following is a list of configuration options for Loki.
|
||||
|
||||
The first option to configure is the name of your connection:
|
||||
|
||||
- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: loki-1, loki_logs.
|
||||
|
||||
- **Default** - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source.
|
||||
|
||||
### HTTP section
|
||||
|
||||
- **URL** - The URL of your Loki server. Loki uses port 3100. If your Loki server is local, use `http://localhost:3100`. If it is on a server within a network, this is the URL with port where you are running Loki. Example: `http://loki.example.orgname:3100`.
|
||||
|
||||
- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default.
|
||||
|
||||
- **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you.
|
||||
|
||||
### Auth section
|
||||
|
||||
There are several authentication methods you can choose in the Authentication section.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki see [Grafana Loki configuration parameters](/docs/loki/latest/configuration/).
|
||||
{{< /admonition >}}
|
||||
|
||||
- **Basic authentication** - The most common authentication method. Use your `data source` user name and `data source` password to connect.
|
||||
|
||||
- **With credentials** - Toggle on to enable credentials such as cookies or auth headers to be sent with cross-site requests.
|
||||
|
||||
- **TLS client authentication** - Toggle on to use client authentication. When enabled, add the `Server name`, `Client cert` and `Client key`. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server.
|
||||
|
||||
- **With CA cert** - Authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file.
|
||||
|
||||
- **Skip TLS verify** - Toggle on to bypass TLS certificate validation.
|
||||
|
||||
- **Forward OAuth identity** - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source.
|
||||
|
||||
### Custom HTTP headers
|
||||
|
||||
- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Loki instance.
|
||||
|
||||
- **Value** - The value of the header.
|
||||
|
||||
### Alerting
|
||||
|
||||
- **Manage alert rules in Alerting UI** - Toggle on to manage alert rules for the Loki data source. To manage other alerting resources add an `Alertmanager` data source.
|
||||
|
||||
### Queries
|
||||
|
||||
- **Maximum lines** - Sets the maximum number of log lines returned by Loki. Increase the limit to have a bigger results set for ad-hoc analysis. Decrease the limit if your browser is sluggish when displaying log results. The default is `1000`.
|
||||
|
||||
<!-- {{< admonition type="note" >}}
|
||||
To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
|
||||
{{< /admonition >}} -->
|
||||
|
||||
### Derived fields
|
||||
|
||||
Derived Fields are used to extract new fields from your logs and create a link from the value of the field.
|
||||
|
||||
For example, you can link to your tracing backend directly from your logs, or link to a user profile page if the log line contains a corresponding `userId`.
|
||||
These links appear in the [log details](ref:log-details).
|
||||
|
||||
You can add multiple derived fields.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
If you use Grafana Cloud, you can request modifications to this feature by clicking **Open a Support Ticket** from the Grafana Cloud Portal.
|
||||
{{< /admonition >}}
|
||||
|
||||
Each derived field consists of the following:
|
||||
|
||||
- **Name** - Sets the field name. Displayed as a label in the log details.
|
||||
|
||||
- **Type** - Defines the type of the derived field. It can be either:
|
||||
|
||||
{{< admonition type="caution" >}}
|
||||
Using complex regular expressions in either type can impact browser performance when processing large volumes of logs. Consider using simpler patterns when possible.
|
||||
{{< /admonition >}}
|
||||
|
||||
- **Regex**: A regular expression to parse a part of the log message and capture it as the value of the new field. Can contain only one capture group.
|
||||
|
||||
- **Label**: A label from the selected log line. This can be any type of label - indexed, parsed or structured metadata. When using this type, the input will match as a regular expression against label keys, allowing you to match variations like `traceid` and `trace_id` with a single regex pattern like `trace[_]?id`. The value of the matched label will be used as the value of the derived field.
|
||||
|
||||
- **URL/query** Sets the full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro.
|
||||
|
||||
- **URL Label** - Sets a custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source.
|
||||
|
||||
- **Internal link** - Toggle on to define an internal link. For internal links, you can select the target data source from a selector. This supports only tracing data sources.
|
||||
|
||||
- **Open in new tab** - Toggle on to open the link in a new tab or window.
|
||||
|
||||
- **Show example log message** - Click to paste an example log line to test the regular expression of your derived fields.
|
||||
|
||||
Click **Save & test** to test your connection.
|
||||
|
||||
#### Troubleshoot interpolation
|
||||
|
||||
You can use a debug section to see what your fields extract and how the URL is interpolated.
|
||||
Select **Show example log message** to display a text area where you can enter a log message.
|
||||
|
||||
{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debugging" >}}
|
||||
|
||||
The new field with the link shown in log details:
|
||||
|
||||
{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}}
|
||||
363
docs/sources/datasources/loki/configure/index.md
Normal file
363
docs/sources/datasources/loki/configure/index.md
Normal file
@@ -0,0 +1,363 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/loki/configure/
|
||||
description: Configure the Loki data source
|
||||
keywords:
|
||||
- grafana
|
||||
- loki
|
||||
- logging
|
||||
- guide
|
||||
- data source
|
||||
menuTitle: Configure
|
||||
title: Configure the Loki data source
|
||||
weight: 200
|
||||
refs:
|
||||
log-details:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/logs-integration/#labels-and-detected-fields
|
||||
alerting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/alerting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/alerting-and-irm/alerting/
|
||||
private-data-source-connect:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
configure-pdc:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc
|
||||
provisioning-data-sources:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/provisioning/#data-sources
|
||||
data-source-management:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/administration/data-source-management/
|
||||
loki-query-editor:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/
|
||||
loki-template-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
---
|
||||
|
||||
# Configure the Loki data source
|
||||
|
||||
This document provides instructions for configuring the Loki data source and explains available configuration options. For general information about data sources, refer to [Data source management](ref:data-source-management).
|
||||
|
||||
Grafana ships with built-in support for [Loki](https://grafana.com/docs/loki/latest/), an open-source log aggregation system by Grafana Labs. If you are new to Loki, the following documentation will help you get started:
|
||||
|
||||
- [Getting started](https://grafana.com/docs/loki/latest/get-started/)
|
||||
- [Best practices](https://grafana.com/docs/loki/latest/best-practices/#best-practices)
|
||||
|
||||
## Before you begin
|
||||
|
||||
Before configuring the Loki data source, ensure you have the following:
|
||||
|
||||
- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system or [using Terraform](#provision-the-data-source-using-terraform).
|
||||
|
||||
- **Loki instance:** You need a running Loki instance and its URL. If you don't have one, refer to the [Loki installation documentation](https://grafana.com/docs/loki/latest/setup/install/).
|
||||
|
||||
- **Authentication details (if applicable):** If your Loki instance requires authentication, gather the necessary credentials such as username and password for basic authentication, or any required certificates for TLS authentication.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
The Loki data source plugin is built into Grafana. No additional installation is required.
|
||||
{{< /admonition >}}
|
||||
|
||||
## Add the Loki data source
|
||||
|
||||
To add the Loki data source, complete the following steps:
|
||||
|
||||
1. Click **Connections** in the left-side menu.
|
||||
1. Under **Connections**, click **Add new connection**.
|
||||
1. Enter `Loki` in the search bar.
|
||||
1. Select **Loki data source**.
|
||||
1. Click **Create a Loki data source** in the upper right.
|
||||
|
||||
You are taken to the **Settings** tab where you will set up your Loki configuration.
|
||||
|
||||
## Configure Loki using the UI
|
||||
|
||||
The following are the configuration options for Loki.
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | ---------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Name** | The data source name. This is how you refer to the data source in panels and queries. Examples: `loki-1`, `loki_logs`. |
|
||||
| **Default** | Toggle to set this data source as the default. When enabled, new panels automatically use this data source. |
|
||||
|
||||
### Connection section
|
||||
|
||||
| Name | Description |
|
||||
| ------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **URL** | The URL of your Loki server, including the port. The default Loki port is `3100`. Examples: `http://localhost:3100`, `http://loki.example.org:3100`. |
|
||||
|
||||
### Authentication section
|
||||
|
||||
Select an authentication method from the **Authentication** dropdown.
|
||||
|
||||
| Setting | Description |
|
||||
| -------------------------- | ---------------------------------------------------------------------------------------------------------- |
|
||||
| **No authentication** | No authentication is required to access the data source. |
|
||||
| **Basic authentication** | Authenticate using a username and password. Enter the credentials in the **User** and **Password** fields. |
|
||||
| **Forward OAuth identity** | Forward the OAuth access token (and the OIDC ID token if available) of the user querying the data source. |
|
||||
|
||||
### TLS settings
|
||||
|
||||
Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For more information on setting up TLS encryption with Loki, refer to [Grafana Loki configuration parameters](https://grafana.com/docs/loki/latest/configuration/).
|
||||
|
||||
| Setting | Description |
|
||||
| ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Add self-signed certificate** | Enable to add a self-signed CA certificate. When enabled, enter the certificate in the **CA Certificate** field. The certificate must begin with `-----BEGIN CERTIFICATE-----`. |
|
||||
| **TLS Client Authentication** | Enable to use client certificate authentication. When enabled, enter the **ServerName** (for example, `domain.example.com`), **Client Certificate** (begins with `-----BEGIN CERTIFICATE-----`), and **Client Key** (begins with `-----BEGIN RSA PRIVATE KEY-----`). |
|
||||
| **Skip TLS certificate validation** | Enable to bypass TLS certificate validation. Use this option only for testing or when connecting to Loki instances with self-signed certificates. |
|
||||
|
||||
### HTTP headers
|
||||
|
||||
Use HTTP headers to pass along additional context and metadata about the request/response.
|
||||
|
||||
| Setting | Description |
|
||||
| ---------- | -------------------------------------------------------------- |
|
||||
| **Header** | The name of the custom header. For example, `X-Custom-Header`. |
|
||||
| **Value** | The value of the custom header. For example, `Header value`. |
|
||||
|
||||
Click **+ Add another header** to add additional headers.
|
||||
|
||||
## Additional settings
|
||||
|
||||
Additional settings are optional settings that you can configure for more control over your data source.
|
||||
|
||||
### Advanced HTTP settings
|
||||
|
||||
| Setting | Description |
|
||||
| ------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Allowed cookies** | Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default. |
|
||||
| **Timeout** | The HTTP request timeout in seconds. If not set, the default Grafana timeout is used. |
|
||||
|
||||
### Alerting
|
||||
|
||||
Manage alert rules for the Loki data source. For more information, refer to [Alerting](ref:alerting).
|
||||
|
||||
| Setting | Description |
|
||||
| ------------------------------------- | ---------------------------------------------------------------------------------- |
|
||||
| **Manage alert rules in Alerting UI** | Toggle to manage alert rules for this Loki data source in the Grafana Alerting UI. |
|
||||
|
||||
### Queries
|
||||
|
||||
Configure options to customize your querying experience.
|
||||
|
||||
| Setting | Description |
|
||||
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Maximum lines** | The maximum number of log lines returned by Loki. The default is `1000`. Increase for larger result sets during ad-hoc analysis. Decrease if your browser is sluggish when displaying log results. |
|
||||
|
||||
### Derived fields
|
||||
|
||||
Derived fields can be used to extract new fields from a log message and create a link from its value. For example, you can link to your tracing backend directly from your logs. These links appear in the [log details](ref:log-details).
|
||||
|
||||
Click **+ Add** to add a derived field. Each derived field has the following settings:
|
||||
|
||||
| Setting | Description |
|
||||
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Name** | The field name. Displayed as a label in the log details. |
|
||||
| **Type** | The type of derived field. Select **Regex in log line** to extract values using a regular expression, or **Label** to use an existing label value. |
|
||||
| **Regex** | A regular expression to parse a part of the log message and capture it as the value of the new field. Can contain only one capture group. |
|
||||
| **URL** | The full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro. For example, `http://example.com/${__value.raw}`. |
|
||||
| **URL Label** | A custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source. |
|
||||
| **Internal link** | Toggle to define an internal link. When enabled, you can select the target data source from a selector. This supports only tracing data sources. |
|
||||
| **Open in new tab** | Toggle to open the link in a new browser tab or window. |
|
||||
|
||||
{{< admonition type="caution" >}}
|
||||
Using complex regular expressions can impact browser performance when processing large volumes of logs. Consider using simpler patterns when possible.
|
||||
{{< /admonition >}}
|
||||
|
||||
#### Test derived fields
|
||||
|
||||
To test your derived field configuration:
|
||||
|
||||
1. Click **Show example log message** to display the debug section.
|
||||
1. In the **Debug log message** field, paste an example log line to test the regular expressions of your derived fields.
|
||||
1. Verify that the field extracts the expected value and the URL is interpolated correctly.
|
||||
|
||||
### Private data source connect
|
||||
|
||||
_Only for Grafana Cloud users._
|
||||
|
||||
Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC, refer to [Private data source connect (PDC)](ref:private-data-source-connect) and [Configure Grafana private data source connect (PDC)](ref:configure-pdc) for instructions on setting up a PDC connection.
|
||||
|
||||
Click **Manage private data source connect** to open your PDC connection page and view your configuration details.
|
||||
|
||||
## Verify the connection
|
||||
|
||||
After configuring the data source, click **Save & test** to save your settings and verify the connection. A successful connection displays the following message:
|
||||
|
||||
**Data source successfully connected.**
|
||||
|
||||
If the test fails, verify:
|
||||
|
||||
- The Loki URL is correct and accessible from the Grafana server.
|
||||
- Any required authentication credentials are correct.
|
||||
- Network connectivity and firewall rules allow the connection.
|
||||
- TLS certificates are valid (if using HTTPS).
|
||||
|
||||
## Provision the data source
|
||||
|
||||
You can define and configure the data source in YAML files as part of the Grafana provisioning system.
|
||||
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources).
|
||||
|
||||
### Provisioning examples
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://localhost:3100
|
||||
jsonData:
|
||||
timeout: 60
|
||||
maxLines: 1000
|
||||
```
|
||||
|
||||
**Using basic authorization and a derived field:**
|
||||
|
||||
You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables:
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://localhost:3100
|
||||
basicAuth: true
|
||||
basicAuthUser: my_user
|
||||
jsonData:
|
||||
maxLines: 1000
|
||||
derivedFields:
|
||||
# Field with internal link pointing to data source in Grafana.
|
||||
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
|
||||
- datasourceUid: my_jaeger_uid
|
||||
matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
# url will be interpreted as query for the datasource
|
||||
url: '$${__value.raw}'
|
||||
# optional for URL Label to set a custom display label for the link.
|
||||
urlDisplayLabel: 'View Trace'
|
||||
|
||||
# Field with external link.
|
||||
- matcherRegex: "traceID=(\\w+)"
|
||||
name: TraceID
|
||||
url: 'http://localhost:16686/trace/$${__value.raw}'
|
||||
secureJsonData:
|
||||
basicAuthPassword: test_password
|
||||
```
|
||||
|
||||
**Using a Jaeger data source:**
|
||||
|
||||
In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value.
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Jaeger
|
||||
type: jaeger
|
||||
url: http://jaeger-tracing-query:16686/
|
||||
access: proxy
|
||||
# UID should match the datasourceUid in derivedFields.
|
||||
uid: my_jaeger_uid
|
||||
```
|
||||
|
||||
## Provision the data source using Terraform
|
||||
|
||||
You can provision the Loki data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs).
|
||||
|
||||
For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation.
|
||||
|
||||
### Basic Terraform example
|
||||
|
||||
The following example creates a basic Loki data source:
|
||||
|
||||
```hcl
|
||||
resource "grafana_data_source" "loki" {
|
||||
name = "Loki"
|
||||
type = "loki"
|
||||
url = "http://localhost:3100"
|
||||
|
||||
json_data_encoded = jsonencode({
|
||||
maxLines = 1000
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Terraform example with derived fields
|
||||
|
||||
The following example creates a Loki data source with a derived field that links to a Jaeger data source for trace correlation:
|
||||
|
||||
```hcl
|
||||
resource "grafana_data_source" "loki_with_tracing" {
|
||||
name = "Loki"
|
||||
type = "loki"
|
||||
url = "http://localhost:3100"
|
||||
|
||||
json_data_encoded = jsonencode({
|
||||
maxLines = 1000
|
||||
derivedFields = [
|
||||
{
|
||||
datasourceUid = grafana_data_source.jaeger.uid
|
||||
matcherRegex = "traceID=(\\w+)"
|
||||
name = "TraceID"
|
||||
url = "$${__value.raw}"
|
||||
urlDisplayLabel = "View Trace"
|
||||
}
|
||||
]
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
### Terraform example with basic authentication
|
||||
|
||||
The following example includes basic authentication:
|
||||
|
||||
```hcl
|
||||
resource "grafana_data_source" "loki_auth" {
|
||||
name = "Loki"
|
||||
type = "loki"
|
||||
url = "http://localhost:3100"
|
||||
|
||||
basic_auth_enabled = true
|
||||
basic_auth_username = "loki_user"
|
||||
|
||||
secure_json_data_encoded = jsonencode({
|
||||
basicAuthPassword = var.loki_password
|
||||
})
|
||||
|
||||
json_data_encoded = jsonencode({
|
||||
maxLines = 1000
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source).
|
||||
|
||||
## Next steps
|
||||
|
||||
After configuring your Loki data source, explore these resources:
|
||||
|
||||
- [Query the Loki data source](ref:loki-query-editor) to learn how to build LogQL queries in Grafana
|
||||
- [Use template variables](ref:loki-template-variables) to create dynamic, reusable dashboards
|
||||
- [LogQL documentation](https://grafana.com/docs/loki/latest/query/) to learn more about the Loki query language
|
||||
@@ -16,11 +16,6 @@ menuTitle: Query editor
|
||||
title: Loki query editor
|
||||
weight: 300
|
||||
refs:
|
||||
annotate-visualizations:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/build-dashboards/annotate-visualizations/
|
||||
logs:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/panels-visualizations/visualizations/logs/
|
||||
@@ -36,231 +31,247 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/explore/
|
||||
template-variables:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/template-variables/
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
loki-troubleshooting:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/troubleshooting/
|
||||
---
|
||||
|
||||
# Loki query editor
|
||||
|
||||
The Loki data source's query editor helps you create [log](#create-a-log-query) and [metric](#create-a-metric-query) queries that use Loki's query language, [LogQL](/docs/loki/latest/logql/).
|
||||
The Loki data source query editor helps you create [log](#create-a-log-query) and [metric](#create-a-metric-query) queries using [LogQL](https://grafana.com/docs/loki/latest/logql/), Loki's query language.
|
||||
|
||||
You can query and display log data from Loki in [Explore](ref:explore) and in dashboards using the [Logs panel](ref:logs).
|
||||
|
||||
For general documentation on querying data sources in Grafana, refer to [Query and transform data](ref:query-transform-data).
|
||||
|
||||
## Before you begin
|
||||
|
||||
- [Configure the Loki data source](ref:configure-loki).
|
||||
- Familiarize yourself with [LogQL](https://grafana.com/docs/loki/latest/logql/).
|
||||
|
||||
## Choose a query editing mode
|
||||
|
||||
The Loki query editor has two modes:
|
||||
|
||||
- [Builder mode](#builder-mode), which provides a visual query designer.
|
||||
- [Code mode](#code-mode), which provides a feature-rich editor for writing queries.
|
||||
- **Builder mode** - Build queries using a visual interface without manually entering LogQL. Best for users new to Loki and LogQL.
|
||||
- **Code mode** - Write queries using a text editor with autocompletion, syntax highlighting, and query validation.
|
||||
|
||||
To switch between the editor modes, select the corresponding **Builder** and **Code** tabs.
|
||||
To switch between modes, select the **Builder** or **Code** tab at the top of the editor.
|
||||
|
||||
To run a query, select **Run queries** located at the top of the editor.
|
||||
Both modes are synchronized, so you can switch between them without losing your work. However, Builder mode doesn't support some complex queries. When switching from Code mode to Builder mode with an unsupported query, the editor displays a warning explaining which parts of the query might be lost.
|
||||
|
||||
{{< admonition type="note" >}}
|
||||
To run Loki queries in [Explore](ref:explore), select **Run query**.
|
||||
{{< /admonition >}}
|
||||
## Toolbar features
|
||||
|
||||
Each mode is synchronized, so you can switch between them without losing your work, although there are some limitations. Builder mode doesn't support some complex queries.
|
||||
When you switch from Code mode to Builder mode with such a query, the editor displays a warning message that explains how you might lose parts of the query if you continue.
|
||||
You can then decide whether you still want to switch to Builder mode.
|
||||
The query editor toolbar provides features available in both Builder and Code mode.
|
||||
|
||||
You can also augment queries by using [template variables](../template-variables/).
|
||||
### Kick start your query
|
||||
|
||||
## Toolbar elements
|
||||
|
||||
The query editor toolbar contains the following elements:
|
||||
|
||||
- **Kick start your query** - Click to see a list of queries that help you quickly get started creating LogQL queries. You can then continue to complete your query.
|
||||
|
||||
These include:
|
||||
Click **Kick start your query** to see a list of example queries that help you get started quickly. These include:
|
||||
|
||||
- Log query starters
|
||||
- Metric query starters
|
||||
|
||||
Click the arrow next to each to see available query options.
|
||||
Click the arrow next to each category to see available query templates. Selecting a template populates the query editor with a starting query you can customize.
|
||||
|
||||
- **Label browser** - Use the Loki label browser to navigate through your labels and values, and build queries.
|
||||
### Label browser
|
||||
|
||||
To navigate Loki and build a query:
|
||||
Use the label browser to explore available labels and values in your Loki instance:
|
||||
|
||||
1. Choose labels to locate.
|
||||
1. Search for the values of your selected labels.
|
||||
1. Click **Label browser** in the toolbar.
|
||||
1. Select labels to filter.
|
||||
1. Search for values using the search field, which supports fuzzy matching.
|
||||
|
||||
The search field supports fuzzy search, and the label browser also supports faceting to list only possible label combinations.
|
||||
The label browser supports faceting to show only valid label combinations.
|
||||
|
||||
1. Select the **Show logs** button to display log lines based on the selected labels, or select the **Show logs rate** button to show the rate based on metrics such as requests per second. Additionally, you can validate the selector by clicking the **Validate selector** button. Click **Clear** to start from the beginning.
|
||||
Click **Show logs** to display log lines based on the selected labels, or **Show logs rate** to show a rate metric. Use **Validate selector** to check your selection, or **Clear** to start over.
|
||||
|
||||
{{< figure src="/static/img/docs/explore/Loki_label_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki label browser" >}}
|
||||
|
||||
- **Explain query** - Toggle to display a step-by-step explanation of all query components and operations.
|
||||
### Explain query
|
||||
|
||||
{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain results" >}}
|
||||
Toggle **Explain query** to display a step-by-step explanation of all query components and operations. This helps you understand how your query works and learn LogQL syntax.
|
||||
|
||||
- **Builder/Code** - Click the corresponding **Builder** or **Code** tab on the toolbar to select an editor mode.
|
||||
{{< figure src="/static/img/docs/prometheus/explain-results.png" max-width="500px" class="docs-image--no-shadow" caption="Explain query results" >}}
|
||||
|
||||
## Builder mode
|
||||
## Build a query in Builder mode
|
||||
|
||||
Builder mode helps you build queries using a visual interface without needing to manually enter LogQL. This option is best for users who have limited or no previous experience working with Loki and LogQL.
|
||||
Builder mode provides a visual interface for constructing LogQL queries without writing code.
|
||||
|
||||
### Label filters
|
||||
### Select labels
|
||||
|
||||
Select labels and their values from the dropdown list.
|
||||
When you select a label, Grafana retrieves available values from the server.
|
||||
Start by selecting labels to filter your log streams:
|
||||
|
||||
Use the `+` button to add a label and the `x` button to remove a label. You can add multiple labels.
|
||||
1. Select a label from the **Label** dropdown.
|
||||
1. Choose a comparison operator:
|
||||
- `=` - equals
|
||||
- `!=` - does not equal
|
||||
- `=~` - matches regex
|
||||
- `!~` - does not match regex
|
||||
1. Select a value from the **Value** dropdown, which displays available values for the selected label.
|
||||
|
||||
Select comparison operators from the following options:
|
||||
Use the `+` button to add additional label filters and the `x` button to remove them.
|
||||
|
||||
- `=` - equal to
|
||||
- `!=` - is not equal
|
||||
- `=~` - matches regex
|
||||
- `!~` - does not match regex
|
||||
### Add operations
|
||||
|
||||
Select values by using the dropdown, which displays all possible values based on the label selected.
|
||||
Select the **+ Operations** button to add operations to your query. The query editor groups operations into the following categories:
|
||||
|
||||
### Operations
|
||||
- **Aggregations** - refer to [Built-in aggregation operators](https://grafana.com/docs/loki/latest/logql/metric_queries/#built-in-aggregation-operators)
|
||||
- **Range functions** - refer to [Range Vector aggregation](https://grafana.com/docs/loki/latest/logql/metric_queries/#range-vector-aggregation)
|
||||
- **Formats** - refer to [Log queries](https://grafana.com/docs/loki/latest/logql/log_queries/#log-queries)
|
||||
- **Binary operations** - refer to [Binary operators](https://grafana.com/docs/loki/latest/logql/#binary-operators)
|
||||
- **Label filters** - refer to [Label filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#label-filter-expression)
|
||||
- **Line filters** - refer to [Line filter expression](https://grafana.com/docs/loki/latest/logql/log_queries/#line-filter-expression)
|
||||
|
||||
Select the `+ Operations` button to add operations to your query.
|
||||
The query editor groups operations into related sections, and you can type while the operations dropdown is open to search and filter the list.
|
||||
You can type while the operations dropdown is open to search and filter the list.
|
||||
|
||||
The query editor displays a query's operations as boxes in the operations section.
|
||||
Each operation's header displays its name, and additional action buttons appear when you hover your cursor over the header:
|
||||
Each operation appears as a box in the query editor. Hover over an operation's header to reveal action buttons:
|
||||
|
||||
| Button | Action |
|
||||
| ----------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_replace.png" class="docs-image--no-shadow" max-width="30px" >}} | Replaces the operation with different operation of the same type. |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_description.png" class="docs-image--no-shadow" max-width="30px" >}} | Opens the operation's description tooltip. |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_remove.png" class="docs-image--no-shadow" max-width="30px" >}} | Removes the operation. |
|
||||
| Button | Action |
|
||||
| ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------ |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_replace.png" class="docs-image--no-shadow" max-width="30px" >}} | Replace the operation with a different operation of the same type. |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_description.png" class="docs-image--no-shadow" max-width="30px" >}} | Open the operation's description tooltip. |
|
||||
| {{< figure src="/static/img/docs/v95/loki_operation_remove.png" class="docs-image--no-shadow" max-width="30px" >}} | Remove the operation. |
|
||||
|
||||
The query editor groups operations into the following sections:
|
||||
Some operations only make sense in a specific order. If adding an operation would result in an invalid query, the editor automatically places it in the correct position. To re-order operations manually, drag the operation box by its name and drop it in the desired location.
|
||||
|
||||
- Aggregations - see [Built-in aggregation operators](/docs/loki/latest/logql/metric_queries/#built-in-aggregation-operators)
|
||||
- Range functions - see [Range Vector aggregation](/docs/loki/latest/logql/metric_queries/#range-vector-aggregation)
|
||||
- Formats - see [Log queries](/docs/loki/latest/logql/log_queries/#log-queries)
|
||||
- Binary operations - see [Binary operators](/docs/loki/latest/logql/#binary-operators)
|
||||
- Label filters - see [Label filter expression](/docs/loki/latest/logql/log_queries/#label-filter-expression)
|
||||
- Line filters - see [Line filter expression](/docs/loki/latest/logql/log_queries/#label-filter-expression)
|
||||
For more information, refer to [Order of operations](https://grafana.com/docs/loki/latest/logql/#order-of-operations).
|
||||
|
||||
Some operations make sense only when used in a specific order. If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.
|
||||
To re-order operations manually, drag the operation box by its name and drop it into the desired place. For additional information see [Order of operations](/docs/loki/latest/logql/#order-of-operations).
|
||||
### Query preview
|
||||
|
||||
As you build your query, the editor displays a visual preview of the query structure. Each step is numbered and includes a description:
|
||||
|
||||
- **Step 1** typically shows your label selector (for example, `{}` with "Fetch all log lines matching label filters")
|
||||
- **Subsequent steps** show operations you've added (for example, `|= ""` with "Return log lines that contain string")
|
||||
|
||||
The raw LogQL query is displayed at the bottom of the query editor, showing the complete syntax that will be executed.
|
||||
|
||||
### Hints
|
||||
|
||||
In same cases the query editor can detect which operations would be most appropriate for a selected log stream. In such cases it will show a hint next to the `+ Operations` button. Click on the hint to add the operations to your query.
|
||||
The query editor can detect which operations would be most appropriate for a selected log stream. When available, a hint appears next to the **+ Operations** button. Click the hint to add the suggested operations to your query.
|
||||
|
||||
## Code mode
|
||||
## Write a query in Code mode
|
||||
|
||||
In **Code mode**, you can write complex queries using a text editor with autocompletion feature, syntax highlighting, and query validation.
|
||||
It also contains a [label browser](#label-browser) to further help you write queries.
|
||||
Code mode provides a text editor for writing LogQL queries directly. This mode is ideal for complex queries or users familiar with LogQL syntax.
|
||||
|
||||
For more information about Loki's query language, refer to the [Loki documentation](/docs/loki/latest/logql/).
|
||||
### Autocompletion
|
||||
|
||||
### Use autocompletion
|
||||
Autocompletion works automatically as you type. The editor can autocomplete:
|
||||
|
||||
Code mode's autocompletion feature works automatically while typing.
|
||||
- Static functions, aggregations, and keywords
|
||||
- Dynamic items like labels and label values
|
||||
|
||||
The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels.
|
||||
The autocompletion dropdown includes documentation for the suggested items where available.
|
||||
The autocompletion dropdown includes documentation for suggested items where available.
|
||||
|
||||
## Options
|
||||
## Configure query options
|
||||
|
||||
The following options are the same for both **Builder** and **Code** mode:
|
||||
The following options are available in both Builder and Code mode. Expand the **Options** section to configure them.
|
||||
|
||||
- **Legend** - Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`.
|
||||
| Option | Description |
|
||||
| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Legend** | Controls the time series name using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`. |
|
||||
| **Type** | Selects the query type. `instant` queries a single point in time (uses the "To" time from the time range). `range` queries over the selected time range. |
|
||||
| **Line limit** | Defines the maximum number of log lines returned by a query. Default is `1000`. |
|
||||
| **Direction** | Determines the search order. **Backward** searches from the end of the time range (default). **Forward** searches from the beginning. |
|
||||
| **Step** | Sets the step parameter for metric queries. Default is `$__auto`, calculated using the time range and graph width. |
|
||||
|
||||
- **Type** - Selects the query type to run. The `instant` type queries against a single point in time. We use the "To" time from the time range. The `range` type queries over the selected range of time.
|
||||
### Query stats
|
||||
|
||||
- **Line limit** -Defines the upper limit for the number of log lines returned by a query. The default is `1000`
|
||||
The Options section displays query statistics to help you estimate the size and cost of your query before running it. Stats include:
|
||||
|
||||
- **Direction** - Determines the search order. **Backward** is a backward search starting at the end of the time range. **Forward** is a forward search starting at the beginning of the time range. The default is **Backward**
|
||||
- **Streams** - Number of log streams matching your label selectors
|
||||
- **Chunks** - Number of data chunks to be scanned
|
||||
- **Bytes** - Estimated data size
|
||||
- **Entries** - Estimated number of log entries
|
||||
|
||||
- **Step** Sets the step parameter of Loki metrics queries. The default value equals to the value of `$__auto` variable, which is calculated using the time range and the width of the graph (the number of pixels).
|
||||
These statistics update automatically as you build your query and can help you optimize queries to reduce load on your Loki instance.
|
||||
|
||||
## Run a query
|
||||
|
||||
To execute your query, click **Run queries** at the top of the query editor. The results display in the visualization panel below the editor.
|
||||
|
||||
In Explore, you can also press `Shift+Enter` to run the query.
|
||||
|
||||
## Create a log query
|
||||
|
||||
Loki log queries return the contents of the log lines.
|
||||
You can query and display log data from Loki via [Explore](ref:explore), and with the [Logs panel](ref:logs) in dashboards.
|
||||
Log queries return the contents of log lines. These are the most common type of Loki query.
|
||||
|
||||
To display the results of a log query, select the Loki data source, then enter a LogQL query.
|
||||
To create a log query:
|
||||
|
||||
For more information about log queries and LogQL, refer to the [Loki log queries documentation](/docs/loki/latest/logql/log_queries/).
|
||||
1. Select labels to filter your log streams.
|
||||
1. Optionally add line filters to search for specific text patterns.
|
||||
1. Optionally add parsers (like `json` or `logfmt`) to extract fields from log lines.
|
||||
1. Click **Run queries** to execute the query.
|
||||
|
||||
For more information about log queries and LogQL, refer to the [Loki log queries documentation](https://grafana.com/docs/loki/latest/logql/log_queries/).
|
||||
|
||||
### Show log context
|
||||
|
||||
In Explore, you can can retrieve the context surrounding your log results by clicking the `Show Context` button. You'll be able to investigate the logs from the same log stream that came before and after the log message you're interested in.
|
||||
In Explore, click **Show Context** on any log line to view the surrounding logs from the same log stream.
|
||||
|
||||
The initial log context query is created from all labels defining the stream for the selected log line. You can use the log context query editor to widen the search by removing one or more of the label filters from log stream. Additionally, if you used a parser in your original query, you can refine your search by using extracted labels filters.
|
||||
The initial context query uses all labels from the selected log line. You can widen the search by removing label filters in the log context query editor. If your original query used a parser, you can also refine the search using extracted label filters.
|
||||
|
||||
To reduce the repetition of selecting and removing the same labels when examining multiple log context windows, Grafana stores your selected labels and applies them to each open context window. This lets you seamlessly navigate through various log context windows without having to reapply your filters.
|
||||
Grafana stores your label selections and applies them to each context window you open, so you don't need to reapply filters when examining multiple log lines.
|
||||
|
||||
To reset filters and use the initial log context query, click the `Revert to initial query` button next to the query preview.
|
||||
|
||||
### Tail live logs
|
||||
|
||||
Loki supports live tailing of logs in real-time in [Explore](ref:explore).
|
||||
|
||||
Live tailing relies on two Websocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server.
|
||||
|
||||
To start tailing logs click the **Live** button in the top right corner of the Explore view.
|
||||
{{< figure src="/static/img/docs/v95/loki_tailing.png" class="docs-image--no-shadow" max-width="80px" >}}
|
||||
|
||||
#### Proxying examples
|
||||
|
||||
If you use reverse proxies, configure them accordingly to use live tailing:
|
||||
|
||||
**Using Apache2 for proxying between the browser and the Grafana server:**
|
||||
|
||||
```
|
||||
ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"
|
||||
```
|
||||
|
||||
**Using NGINX:**
|
||||
|
||||
This example provides a basic NGINX proxy configuration.
|
||||
It assumes that the Grafana server is available at `http://localhost:3000/`, the Loki server is running locally without proxy, and your external site uses HTTPS.
|
||||
If you also host Loki behind an NGINX proxy, repeat the following configuration for Loki.
|
||||
|
||||
In the `http` section of NGINX configuration, add the following map definition:
|
||||
|
||||
```
|
||||
map $http_upgrade $connection_upgrade {
|
||||
default upgrade;
|
||||
'' close;
|
||||
}
|
||||
```
|
||||
|
||||
In your `server` section, add the following configuration:
|
||||
|
||||
```
|
||||
location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
|
||||
proxy_pass http://localhost:3000$request_uri;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto "https";
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:3000/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto "https";
|
||||
}
|
||||
```
|
||||
To reset filters, click **Revert to initial query** next to the query preview.
|
||||
|
||||
## Create a metric query
|
||||
|
||||
You can use LogQL to wrap a log query with functions that create metrics from your logs.
|
||||
Metric queries use LogQL to extract numeric data from logs. You wrap a log query with aggregation functions to create time series data for visualization and alerting.
|
||||
|
||||
For more information about metric queries, refer to the [Loki metric queries documentation](/docs/loki/latest/logql/metric_queries/).
|
||||
### Common metric query patterns
|
||||
|
||||
## Apply annotations
|
||||
| Function | Description | Example |
|
||||
| ------------------- | ----------------------------------------------- | ---------------------------------------------------- |
|
||||
| `rate()` | Calculates the number of log entries per second | `rate({job="app"}[5m])` |
|
||||
| `count_over_time()` | Counts log entries over the specified interval | `count_over_time({job="app"}[1h])` |
|
||||
| `bytes_rate()` | Calculates bytes per second of log entries | `bytes_rate({job="app"}[5m])` |
|
||||
| `sum_over_time()` | Sums extracted numeric values | `sum_over_time({job="app"} \| unwrap duration [5m])` |
|
||||
|
||||
[Annotations](ref:annotate-visualizations) overlay rich event information on top of graphs.
|
||||
You can add annotation queries in the Dashboard menu's Annotations view.
|
||||
### Build a metric query
|
||||
|
||||
You can only use log queries as a source for annotations.
|
||||
Grafana automatically uses log content as annotation text and your log stream labels as tags.
|
||||
You don't need to create any additional mapping.
|
||||
To create a metric query in Builder mode:
|
||||
|
||||
1. Select labels to filter your log streams.
|
||||
1. Click **+ Operations** and select a range function (for example, **Rate**).
|
||||
1. The editor wraps your log selector with the function and adds a time interval.
|
||||
1. Optionally add aggregations like `sum`, `avg`, or `max` to combine results.
|
||||
|
||||
In Code mode, enter the full LogQL expression directly:
|
||||
|
||||
```logql
|
||||
sum(rate({job="app", level="error"}[5m])) by (instance)
|
||||
```
|
||||
|
||||
This query calculates the per-second rate of error logs, then sums the results grouped by instance.
|
||||
|
||||
For more information, refer to the [Loki metric queries documentation](https://grafana.com/docs/loki/latest/logql/metric_queries/).
|
||||
|
||||
## Tail live logs
|
||||
|
||||
Loki supports live tailing of logs in real-time in [Explore](ref:explore).
|
||||
|
||||
To start tailing logs, click the **Live** button in the top right corner of the Explore view.
|
||||
|
||||
{{< figure src="/static/img/docs/v95/loki_tailing.png" class="docs-image--no-shadow" max-width="80px" >}}
|
||||
|
||||
Live tailing relies on two WebSocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server.
|
||||
|
||||
If you use reverse proxies, you may need to configure them to support WebSocket connections. For proxy configuration examples, refer to the [Loki troubleshooting documentation](ref:loki-troubleshooting).
|
||||
|
||||
## Use template variables
|
||||
|
||||
You can use template variables in your queries to create dynamic, reusable dashboards. Template variables appear as dropdown menus at the top of dashboards, allowing users to change query parameters without editing the query directly.
|
||||
|
||||
For information on creating and using template variables with Loki, refer to [Loki template variables](ref:template-variables).
|
||||
|
||||
@@ -38,22 +38,32 @@ refs:
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/variables/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/dashboards/variables/
|
||||
query-editor-options:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/#options
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/query-editor/#options
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
---
|
||||
|
||||
# Loki template variables
|
||||
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
|
||||
Grafana refers to such variables as template variables.
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. Grafana refers to such variables as template variables.
|
||||
|
||||
For an introduction to templating and template variables, refer to the [Templating](ref:variables) and [Add and manage variables](ref:add-template-variables) documentation.
|
||||
|
||||
## Before you begin
|
||||
|
||||
- Ensure you have [configured the Loki data source](ref:configure-loki).
|
||||
- Your Loki instance should have logs with labels that you want to use as variable values.
|
||||
|
||||
## Use query variables
|
||||
|
||||
Variables of the type _Query_ help you query Loki for lists of labels or label values.
|
||||
The Loki data source provides a form to select the type of values expected for a given variable.
|
||||
|
||||
The form has these options:
|
||||
Use _Query_ type variables to dynamically fetch label names or label values from Loki. When you create a query variable with the Loki data source, you can choose what type of data to retrieve:
|
||||
|
||||
| Query type | Example label | Example stream selector | List returned |
|
||||
| ------------ | ------------- | ----------------------- | ---------------------------------------------------------------- |
|
||||
@@ -61,6 +71,26 @@ The form has these options:
|
||||
| Label values | `label` | | Label values for `label`. |
|
||||
| Label values | `label` | `log stream selector` | Label values for `label` in the specified `log stream selector`. |
|
||||
|
||||
### Create a query variable
|
||||
|
||||
To create a query variable for Loki:
|
||||
|
||||
1. Open the dashboard where you want to add the variable.
|
||||
1. Click **Dashboard settings** (gear icon) in the top navigation.
|
||||
1. Select **Variables** in the left menu.
|
||||
1. Click **Add variable**.
|
||||
1. Enter a **Name** for your variable (for example, `job`, `instance`, `level`).
|
||||
1. In the **Type** dropdown, select **Query**.
|
||||
1. In the **Data source** dropdown, select your Loki data source.
|
||||
1. In the **Query type** dropdown, select **Label names** or **Label values**.
|
||||
1. If you selected **Label values**, enter the label name in the **Label** field (for example, `job`).
|
||||
1. Optionally, enter a **Stream selector** to filter the label values (for example, `{namespace="production"}`).
|
||||
1. Click **Run query** to preview the variable values.
|
||||
1. Configure display options such as **Multi-value** or **Include All option** as needed.
|
||||
1. Click **Apply** to save the variable.
|
||||
|
||||
You can now use the variable in your Loki queries with the syntax `${variable_name}`. For example, `{job="$job"}` filters logs by the selected job.
|
||||
|
||||
## Use ad hoc filters
|
||||
|
||||
Loki supports the special **Ad hoc filters** variable type.
|
||||
@@ -68,27 +98,29 @@ You can use this variable type to specify any number of key/value filters, and G
|
||||
|
||||
For more information, refer to [Add ad hoc filters](ref:add-template-variables-add-ad-hoc-filters).
|
||||
|
||||
## Use $\_\_auto variable for Loki metric queries
|
||||
## Use the $\_\_auto variable for Loki metric queries
|
||||
|
||||
Consider using the `$__auto` variable in your Loki metric queries, which will automatically be substituted with the [step value](https://grafana.com/docs/grafana/next/datasources/loki/query-editor/#options) for range queries, and with the selected time range's value (computed from the starting and ending times) for instant queries.
|
||||
Consider using the `$__auto` variable in your Loki metric queries. This variable is automatically substituted with the [step value](ref:query-editor-options) for range queries, and with the selected time range's value (computed from the starting and ending times) for instant queries.
|
||||
|
||||
For more information about variables, refer to [Global built-in variables](ref:add-template-variables-global-variables).
|
||||
|
||||
## Label extraction and indexing in Loki
|
||||
## Extract and index labels in Loki
|
||||
|
||||
Labels play a fundamental role in Loki's log aggregation and querying capabilities. When logs are ingested into Loki, they are often accompanied by metadata called `labels`, which provide contextual information about the log entries. These labels consist of `key-value` pairs and are essential for organizing, filtering, and searching log data efficiently.
|
||||
|
||||
### Label extraction
|
||||
### Extract labels
|
||||
|
||||
During the ingestion process, Loki performs label extraction from log lines. Loki's approach to label extraction is based on `regular expressions`, allowing users to specify custom patterns for parsing log lines and extracting relevant label key-value pairs. This flexibility enables Loki to adapt to various log formats and schemas.
|
||||
|
||||
For example, suppose you have log lines in the following format:
|
||||
|
||||
**2023-07-25 12:34:56 INFO: Request from IP A.B.C.D to endpoint /api/data**
|
||||
```
|
||||
2023-07-25 12:34:56 INFO: Request from IP A.B.C.D to endpoint /api/data
|
||||
```
|
||||
|
||||
To extract labels from this log format, you could define a regular expression to extract the log level ("INFO"), IP address ("A.B.C.D"), and endpoint ("/api/data") as labels. These labels can later be used to filter and aggregate log entries.
|
||||
To extract labels from this log format, you could define a regular expression to extract the log level (`INFO`), IP address (`A.B.C.D`), and endpoint (`/api/data`) as labels. These labels can later be used to filter and aggregate log entries.
|
||||
|
||||
### Indexing labels
|
||||
### Index labels
|
||||
|
||||
Once labels are extracted, Loki efficiently indexes them. The index serves as a lookup mechanism that maps labels to the corresponding log entries. This indexing process enables faster retrieval of logs based on specific label criteria, significantly enhancing query performance.
|
||||
|
||||
@@ -96,6 +128,4 @@ For instance, if you have a label "job" that represents different services in yo
|
||||
|
||||
By effectively extracting and indexing labels, Loki enables users to perform complex and targeted log queries without compromising on query speed and resource consumption.
|
||||
|
||||
Utilizing Loki's indexed labels in combination with Grafana's template variables provides a powerful way to interactively explore and visualize log data. Template variables allow users to create dynamic queries, selecting and filtering logs based on various labels, such as job names, instance IDs, severity levels, or any other contextual information attached to the log entries.
|
||||
|
||||
In conclusion, Loki's label extraction and indexing mechanisms are key components that contribute to its ability to handle vast amounts of log data efficiently. By making use of labels and template variables, users can easily gain valuable insights from their log data and troubleshoot issues effectively.
|
||||
Combining Loki's indexed labels with Grafana template variables provides a powerful way to interactively explore and visualize log data. Template variables let you create dynamic queries that filter logs based on labels such as job names, instance IDs, or severity levels.
|
||||
|
||||
387
docs/sources/datasources/loki/troubleshooting/index.md
Normal file
387
docs/sources/datasources/loki/troubleshooting/index.md
Normal file
@@ -0,0 +1,387 @@
|
||||
---
|
||||
aliases:
|
||||
- ../../data-sources/loki/troubleshooting/
|
||||
description: Troubleshoot issues with the Loki data source in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- loki
|
||||
- troubleshooting
|
||||
- errors
|
||||
- logs
|
||||
labels:
|
||||
products:
|
||||
- cloud
|
||||
- enterprise
|
||||
- oss
|
||||
menuTitle: Troubleshooting
|
||||
title: Troubleshoot issues with the Loki data source
|
||||
weight: 600
|
||||
refs:
|
||||
configure-loki:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/loki/configure/
|
||||
private-data-source-connect:
|
||||
- pattern: /docs/grafana/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
- pattern: /docs/grafana-cloud/
|
||||
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
|
||||
---
|
||||
|
||||
# Troubleshoot issues with the Loki data source
|
||||
|
||||
This document provides troubleshooting information for common errors you may encounter when using the Loki data source in Grafana.
|
||||
|
||||
## Connection errors
|
||||
|
||||
The following errors occur when Grafana cannot establish or maintain a connection to Loki.
|
||||
|
||||
### Unable to connect with Loki
|
||||
|
||||
**Error message:** "Unable to connect with Loki. Please check the server logs for more details."
|
||||
|
||||
**Cause:** Grafana cannot establish a network connection to the Loki server.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the Loki URL is correct in the [data source configuration](ref:configure-loki).
|
||||
1. Check that Loki is running and accessible from the Grafana server.
|
||||
1. Ensure no firewall rules are blocking the connection.
|
||||
1. If using a proxy, verify the proxy settings are correct.
|
||||
1. For Grafana Cloud, ensure you have configured [Private data source connect](ref:private-data-source-connect) if your Loki instance is not publicly accessible.
|
||||
|
||||
### Request timed out
|
||||
|
||||
**Error message:** "context deadline exceeded" or "request timed out"
|
||||
|
||||
**Cause:** The connection to Loki timed out before receiving a response.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Check the network latency between Grafana and Loki.
|
||||
1. Verify Loki is not overloaded or experiencing performance issues.
|
||||
1. Increase the **Timeout** setting in the data source configuration under **Additional settings** > **Advanced HTTP settings**.
|
||||
1. Check if any network devices (load balancers, proxies) are timing out the connection.
|
||||
1. Reduce the time range or complexity of your query.
|
||||
|
||||
### Failed to parse data source URL
|
||||
|
||||
**Error message:** "Failed to parse data source URL"
|
||||
|
||||
**Cause:** The URL entered in the data source configuration is not valid.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the URL format is correct (for example, `http://localhost:3100` or `https://loki.example.com:3100`).
|
||||
1. Ensure the URL includes the protocol (`http://` or `https://`).
|
||||
1. Remove any trailing slashes or invalid characters from the URL.
|
||||
|
||||
## Authentication errors
|
||||
|
||||
The following errors occur when there are issues with authentication credentials or permissions.
|
||||
|
||||
### Unauthorized (401)
|
||||
|
||||
**Error message:** "Status: 401 Unauthorized"
|
||||
|
||||
**Cause:** The authentication credentials are invalid or missing.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the username and password are correct in the data source configuration.
|
||||
1. Check the authentication method matches your Loki configuration.
|
||||
1. If using a bearer token or API key, ensure it is valid and has not expired.
|
||||
1. Verify the credentials have permission to access the Loki API.
|
||||
|
||||
### Forbidden (403)
|
||||
|
||||
**Error message:** "Status: 403 Forbidden"
|
||||
|
||||
**Cause:** The authenticated user does not have permission to access the requested resource.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the user has read access to the log streams you are querying.
|
||||
1. Check Loki's authentication and authorization configuration.
|
||||
1. If using multi-tenancy, ensure the correct tenant ID (X-Scope-OrgID header) is configured.
|
||||
1. Review any access control policies in your Loki deployment.
|
||||
|
||||
## Query errors
|
||||
|
||||
The following errors occur when there are issues with LogQL query syntax or execution.
|
||||
|
||||
### Parse error
|
||||
|
||||
**Error message:** "parse error" or "syntax error"
|
||||
|
||||
**Cause:** The LogQL query contains invalid syntax.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Check the query for typos or missing characters.
|
||||
1. Verify all brackets, braces, and parentheses are properly balanced.
|
||||
1. Ensure label matchers use the correct operators (`=`, `!=`, `=~`, `!~`).
|
||||
1. Verify string values are enclosed in double quotes.
|
||||
1. Refer to the [LogQL documentation](https://grafana.com/docs/loki/latest/query/) for correct syntax.
|
||||
|
||||
**Common syntax issues:**
|
||||
|
||||
| Issue | Incorrect | Correct |
|
||||
| ----------------- | -------------- | -------------- |
|
||||
| Missing quotes | `{job=app}` | `{job="app"}` |
|
||||
| Wrong operator | `{job=="app"}` | `{job="app"}` |
|
||||
| Unbalanced braces | `{job="app"` | `{job="app"}` |
|
||||
| Invalid regex | `{job=~"["}` | `{job=~"\\["}` |
|
||||
|
||||
### Query limits exceeded
|
||||
|
||||
**Error message:** "query returned more than the max number of entries" or "max entries limit exceeded"
|
||||
|
||||
**Cause:** The query returned more log entries than the configured limit allows.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Add more specific label selectors to reduce the number of matching streams.
|
||||
1. Add line filters to narrow down the results (for example, `|= "error"`).
|
||||
1. Reduce the time range of your query.
|
||||
1. Increase the **Maximum lines** setting in the data source configuration.
|
||||
1. If you control the Loki instance, consider adjusting Loki's `max_entries_limit_per_query` setting.
|
||||
|
||||
### Query timeout
|
||||
|
||||
**Error message:** "query timed out"
|
||||
|
||||
**Cause:** The query took longer to execute than the configured timeout.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Simplify the query by adding more selective label matchers.
|
||||
1. Reduce the time range.
|
||||
1. Avoid expensive operations like complex regex patterns on high-cardinality data.
|
||||
1. If you control the Loki instance, check Loki's query timeout settings.
|
||||
|
||||
### Too many outstanding requests
|
||||
|
||||
**Error message:** "too many outstanding requests"
|
||||
|
||||
**Cause:** Loki has reached its limit for concurrent queries.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Wait a moment and retry the query.
|
||||
1. Reduce the number of panels or dashboards querying Loki simultaneously.
|
||||
1. If you control the Loki instance, consider increasing Loki's concurrency limits.
|
||||
|
||||
## Metric query errors
|
||||
|
||||
The following errors occur when using LogQL metric queries.
|
||||
|
||||
### Invalid unwrap expression
|
||||
|
||||
**Error message:** "invalid unwrap expression" or "unwrap: label does not exist"
|
||||
|
||||
**Cause:** The `unwrap` function references a label that doesn't exist or isn't numeric.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the label name in the `unwrap` expression exists in your log data.
|
||||
1. Ensure the label contains numeric values.
|
||||
1. Add a parser stage (`| logfmt`, `| json`, etc.) before `unwrap` to extract the label from log content.
|
||||
|
||||
**Example fix:**
|
||||
|
||||
```logql
|
||||
# Incorrect - label might not exist
|
||||
{job="app"} | unwrap latency
|
||||
|
||||
# Correct - parse the log first
|
||||
{job="app"} | logfmt | unwrap latency
|
||||
```
|
||||
|
||||
### Division by zero
|
||||
|
||||
**Error message:** "division by zero"
|
||||
|
||||
**Cause:** A metric query attempted to divide by zero.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Add conditions to handle cases where the denominator could be zero.
|
||||
1. Use the `or` operator to provide a default value.
|
||||
|
||||
## Common issues
|
||||
|
||||
The following issues don't always produce specific error messages but are commonly encountered.
|
||||
|
||||
### Empty query results
|
||||
|
||||
**Cause:** The query returns no data.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the time range includes data in your Loki instance.
|
||||
1. Check that the label selectors match existing log streams.
|
||||
1. Use the **Label browser** in the query editor to see available labels and values.
|
||||
1. Start with a simple query like `{job="your-job"}` and add filters incrementally.
|
||||
1. Verify logs are being ingested into Loki for the selected time range.
|
||||
|
||||
### Slow query performance
|
||||
|
||||
**Cause:** Queries take a long time to execute.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Add more specific label selectors. Labels are indexed, so filtering by labels is fast.
|
||||
1. Reduce the time range of your query.
|
||||
1. Avoid regex filters on high-volume streams when possible.
|
||||
1. Use line filters (`|=`, `!=`) before expensive regex operations.
|
||||
1. For metric queries, ensure you're using appropriate aggregation intervals.
|
||||
|
||||
**Query optimization tips:**
|
||||
|
||||
| Slow | Fast |
|
||||
| ----------------------------------------- | ------------------------------------------------- |
|
||||
| `{namespace="prod"} \|~ "error.*timeout"` | `{namespace="prod", level="error"} \|= "timeout"` |
|
||||
| `{job=~".+"}` (matches all) | `{job="specific-job"}` |
|
||||
| Wide time range, no filters | Narrow time range with label filters |
|
||||
|
||||
### Labels not appearing in dropdown
|
||||
|
||||
**Cause:** The label browser doesn't show expected labels.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Check that logs with those labels exist in the selected time range.
|
||||
1. Verify the labels are indexed in Loki (not just parsed from log content).
|
||||
1. Refresh the label browser by clicking the refresh button.
|
||||
1. Clear your browser cache and reload the page.
|
||||
|
||||
### Log lines truncated
|
||||
|
||||
**Cause:** Long log lines are cut off in the display.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Click on a log line to expand and view the full content.
|
||||
1. Use the **Wrap lines** option in the logs visualization settings.
|
||||
1. The full log content is always available; only the display is truncated.
|
||||
|
||||
### Derived fields not working
|
||||
|
||||
**Cause:** Derived fields configured in the data source aren't appearing in log details.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the regex pattern in your derived field configuration matches your log format.
|
||||
1. Test the regex in the **Debug** section of the derived fields configuration.
|
||||
1. Ensure the derived field has a valid URL or internal data source configured.
|
||||
1. Check that the log lines contain text matching the regex pattern.
|
||||
|
||||
## Live tailing issues
|
||||
|
||||
The following issues occur when using the live log tailing feature.
|
||||
|
||||
### Live tailing not working
|
||||
|
||||
**Cause:** Live tailing relies on WebSocket connections that may be blocked by proxies or firewalls.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify WebSocket connections are allowed through your network infrastructure.
|
||||
1. Check that your reverse proxy is configured to support WebSocket connections.
|
||||
1. Ensure the Grafana server can establish a WebSocket connection to Loki.
|
||||
|
||||
### Configure reverse proxies for live tailing
|
||||
|
||||
If you use reverse proxies, configure them to support WebSocket connections for live tailing.
|
||||
|
||||
**Apache2 configuration:**
|
||||
|
||||
Add the following to proxy WebSocket connections:
|
||||
|
||||
```apache
|
||||
ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"
|
||||
```
|
||||
|
||||
**NGINX configuration:**
|
||||
|
||||
This example assumes the Grafana server is available at `http://localhost:3000/`, the Loki server is running locally without a proxy, and your external site uses HTTPS. If you also host Loki behind NGINX, repeat this configuration for Loki.
|
||||
|
||||
In the `http` section of your NGINX configuration, add the following map definition:
|
||||
|
||||
```nginx
|
||||
map $http_upgrade $connection_upgrade {
|
||||
default upgrade;
|
||||
'' close;
|
||||
}
|
||||
```
|
||||
|
||||
In your `server` section, add the following configuration:
|
||||
|
||||
```nginx
|
||||
location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
|
||||
proxy_pass http://localhost:3000$request_uri;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto "https";
|
||||
proxy_set_header Connection $connection_upgrade;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
}
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:3000/;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto "https";
|
||||
}
|
||||
```
|
||||
|
||||
## Multi-tenancy issues
|
||||
|
||||
The following errors occur when using Loki in multi-tenant mode.
|
||||
|
||||
### No org id
|
||||
|
||||
**Error message:** "no org id" or "X-Scope-OrgID header required"
|
||||
|
||||
**Cause:** Loki is configured for multi-tenancy but no tenant ID was provided.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Add a custom HTTP header `X-Scope-OrgID` with your tenant ID in the data source configuration.
|
||||
1. Navigate to **Additional settings** > **HTTP headers** and add the header.
|
||||
|
||||
### Tenant not found
|
||||
|
||||
**Error message:** "tenant not found" or "invalid tenant"
|
||||
|
||||
**Cause:** The specified tenant ID doesn't exist or the user doesn't have access.
|
||||
|
||||
**Solution:**
|
||||
|
||||
1. Verify the tenant ID is correct.
|
||||
1. Check that the tenant exists in your Loki deployment.
|
||||
1. Verify the user has permission to access the specified tenant.
|
||||
|
||||
## Get additional help
|
||||
|
||||
If you continue to experience issues:
|
||||
|
||||
- Check the [Grafana community forums](https://community.grafana.com/) for similar issues and solutions.
|
||||
- Review the [Loki documentation](https://grafana.com/docs/loki/latest/) for detailed configuration and query guidance.
|
||||
- Contact Grafana Support if you're an Enterprise, Cloud Pro, or Cloud contracted customer.
|
||||
|
||||
When reporting issues, include the following information:
|
||||
|
||||
- Grafana version
|
||||
- Loki version
|
||||
- Deployment type (self-hosted Loki, Grafana Cloud Logs)
|
||||
- Error messages (redact sensitive information)
|
||||
- Steps to reproduce the issue
|
||||
- Relevant configuration such as data source settings, authentication method, and timeout values (redact credentials)
|
||||
- Sample LogQL query (if applicable, with sensitive data redacted)
|
||||
- Time range of the query
|
||||
- Approximate volume of logs being queried
|
||||
Reference in New Issue
Block a user