public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Refactor data sources content (#57573)

Browse Source 
* Docs: Revise data source index

* Docs: Consolidate data source administration docs

* Docs: Revise panels docs related to data sources

* Docs: Revise Alertmanager data source

* Docs: Reorganize AWS CloudWatch data source docs

* Docs: Reorganize Azure Monitor data source docs

* Docs: Move azuremonitor to azure-monitor

* Docs: Revise Elasticsearch docs

* Docs: Move Elasticsearch index into bundle

* Docs: Revise GCM docs

* Docs: Revise Graphite docs

* Docs: Move Graphite index into bundle

* Docs: Revise InfluxDB docs

* Docs: Revise Jaeger docs

* Docs: Move Jaeger index into bundle

* Docs: Revise Loki docs

* Docs: Move Loki index into bundle

* Docs: Revise MS SQL docs

* Docs: Move MS SQL index into bundle

* Docs: Revise Prometheus docs

* Docs: Move Prometheus index into bundle

* Docs: Revise Tempo docs

* Docs: Move Tempo index into bundle

* Docs: Revise TestData DB docs

* Docs: Move TestData DB index into bundle

* Docs: Revise Zipkin docs

* Docs: Move Zipkin index into bundle

* Docs: Move other data sources' index pages into bundles

* Docs: Revise frontmatter

* Fixing hugo markdown errors

* Docs: Add query editor and template var sections to overview doc

* Docs: Remove CTAs across data source docs

* Docs: Remove CTA

* Docs: Remove CTA

* Docs: Fix links, images, typos, and usage consistency.

* Docs: Fix typos

* Docs: Fix CI issues

* Update docs/sources/datasources/_index.md

Co-authored-by: Torkel Ödegaard <torkel@grafana.com>

* Update docs/sources/datasources/_index.md

Co-authored-by: Torkel Ödegaard <torkel@grafana.com>

* Docs: Fix query editor links

* Update docs/sources/panels-visualizations/_index.md

Co-authored-by: Torkel Ödegaard <torkel@grafana.com>

* Update docs/sources/panels-visualizations/_index.md

Co-authored-by: Torkel Ödegaard <torkel@grafana.com>

* Docs: Rebundle child pages per writers' toolkit

* Docs: Fix prettier for CI

* Docs: Fix relrefs from outside data sources docs

* Docs: Fix broken relrefs within datasources

* Docs: Fix relrefs to data sources docs

* Fixed some more refs

Co-authored-by: Torkel Ödegaard <torkel@grafana.com>
This commit is contained in:
Garrett Guillotte
2022-11-01 10:22:06 -05:00
committed by GitHub
co-authored by Torkel Ödegaard
parent 22648d8581
commit 852d069a3c
77 changed files with 5722 additions and 4410 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/datasources/loki/_index.md
+160
View File
@@ -0,0 +1,160 @@
---
aliases:
- /docs/grafana/latest/features/datasources/loki/
- /docs/grafana/latest/datasources/loki/
- /docs/grafana/latest/data-sources/loki/
description: Guide for using Loki in Grafana
keywords:
- grafana
- loki
- logging
- guide
menuTitle: Loki
title: Loki data source
weight: 800
---
# Loki data source
Grafana ships with built-in support for [Loki](/docs/loki/latest/), an open-source log aggregation system by Grafana Labs.
This topic explains configuring and querying specific to the Loki data source.
For instructions on how to add a data source to Grafana, refer to the [administration documentation]({{< relref "../../administration/data-source-management/" >}}).
Only users with the organization administrator role can add data sources.
Administrators can also [configure the data source via YAML]({{< relref "#provision-the-data-source" >}}) with Grafana's provisioning system.
Once you've added the Loki data source, you can [configure it]({{< relref "#configure-the-data-source" >}}) so that your Grafana instance's users can create queries in its [query editor]({{< relref "./query-editor/" >}}) when they [build dashboards]({{< relref "../../dashboards/build-dashboards/" >}}), use [Explore]({{< relref "../../explore/" >}}), and [annotate visualizations]({{< relref "./query-editor/#apply-annotations" >}}).
## Configure the data source
**To access the data source configuration page:**
1. Hover the cursor over the **Configuration** (gear) icon.
1. Select **Data Sources**.
1. Select the Loki data source.
Set the data source's basic configuration options carefully:
| Name | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name** | Sets the name you use to refer to the data source in panels and queries. |
| **Default** | Sets the data source that's pre-selected for new panels. |
| **URL** | Sets the HTTP protocol, IP, and port of your Loki instance, such as `http://localhost:3100`. |
| **Allowed cookies** | Defines which cookies are forwarded to the data source. Grafana Proxy deletes all other cookies. |
| **Maximum lines** | Sets the upper limit for the number of log lines returned by Loki. Defaults to 1,000. Lower this limit if your browser is sluggish when displaying logs in Explore. |
> **Note:** To troubleshoot configuration and other issues, check the log file located at `/var/log/grafana/grafana.log` on Unix systems, or in `<grafana_install_dir>/data/log` on other platforms and manual installations.
### Configure derived fields
The **Derived Fields** configuration helps you:
- Add fields parsed from the log message
- Add a link that uses the value of the field
For example, you can link to your tracing backend directly from your logs, or link to a user profile page if the log line contains a corresponding userId.
These links appear in the [log details]({{< relref "../../explore/logs-integration/#labels-and-detected-fields" >}}).
> **Note:** If you use Grafana Cloud, you can request modifications to this feature by [opening a support ticket in the Cloud Portal](/profile/org#support).
Each derived field consists of:
| Field name | Description |
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Name** | Sets the field name. Displayed as a label in the log details. |
| **Regex** | Defines a regular expression to evaluate on the log message and capture part of it as the value of the new field. Can contain only one capture group. |
| **URL/query** | Sets the full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro. |
| **URL Label** | _(Optional)_ Sets a custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source. |
| **Internal link** | Defines whether the link is internal or external. For internal links, you can select the target data source from a selector. This supports only tracing data sources. |
#### Troubleshoot interpolation
You can use a debug section to see what your fields extract and how the URL is interpolated.
Select **Show example log message** to display a text area where you can enter a log message.
{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debugging" >}}
The new field with the link shown in log details:
{{< figure src="/static/img/docs/explore/detected-fields-link-7-4.png" max-width="800px" caption="Detected fields link in Explore" >}}
### Provision the data source
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana]({{< relref "../../administration/provisioning/#data-sources" >}}).
#### Provisioning examples
```yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
jsonData:
maxLines: 1000
```
**Using basic authorization and a derived field:**
You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables:
```yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
basicAuth: true
basicAuthUser: my_user
jsonData:
maxLines: 1000
derivedFields:
# Field with internal link pointing to data source in Grafana.
# Right now, Grafana supports only Jaeger and Zipkin data sources as link targets.
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
- datasourceUid: my_jaeger_uid
matcherRegex: "traceID=(\\w+)"
name: TraceID
# url will be interpreted as query for the datasource
url: '$${__value.raw}'
# Field with external link.
- matcherRegex: "traceID=(\\w+)"
name: TraceID
url: 'http://localhost:16686/trace/$${__value.raw}'
secureJsonData:
basicAuthPassword: test_password
```
**Using a Jaeger data source:**
In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value.
```
datasources:
- name: Jaeger
type: jaeger
url: http://jaeger-tracing-query:16686/
access: proxy
# UID should match the datasourceUid in derivedFields.
uid: my_jaeger_uid
```
## Query the data source
The Loki data source's query editor helps you create log and metric queries that use Loki's query language, [LogQL](/docs/loki/latest/logql/).
For details, refer to the [query editor documentation]({{< relref "./query-editor/" >}}).
## Use template variables
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.
For details, see the [template variables documentation]({{< relref "./template-variables/" >}}).
docs/sources/datasources/loki/query-editor/index.md
+224
View File
@@ -0,0 +1,224 @@
---
aliases:
- /docs/grafana/latest/data-sources/loki/query-editor/
description: Guide for using the Loki data source's query editor
keywords:
- grafana
- loki
- logs
- queries
menuTitle: Query editor
title: Loki query editor
weight: 300
---
# Loki query editor
The Loki data source's query editor helps you create [log]({{< relref "#create-a-logs-query" >}}) and [metric]({{< relref "#create-a-metrics-query" >}}) queries that use Loki's query language, [LogQL](/docs/loki/latest/logql/).
This topic explains querying specific to the Loki data source.
For general documentation on querying data sources in Grafana, see [Query and transform data]({{< relref "../../../panels-visualizations/query-transform-data" >}}).
## Choose a query editing mode
You can switch the Loki query editor between two modes:
- [Code mode]({{< relref "#code-mode" >}}), which provides a feature-rich editor for writing queries
- [Builder mode]({{< relref "#builder-mode" >}}), which provides a visual query designer
To switch between the editor modes, select the corresponding **Builder** and **Code** tabs.
To run a query, select **Run queries** located at the top of the editor.
> **Note:** To run Loki queries in [Explore]({{< relref "../../../explore/" >}}), select **Run query**.
Each mode is synchronized with the other modes, so you can switch between them without losing your work, although there are some limitations.
Builder mode doesn't yet support some complex queries.
When you switch from Code mode to Builder mode with such a query, the editor displays a popup that explains how you might lose parts of the query if you continue.
You can then decide whether you still want to switch to Builder mode.
You can also augment queries by using [template variables]({{< relref "./template-variables/" >}}).
## Code mode
In **Code mode**, you can write complex queries using a text editor with autocompletion features and syntax highlighting.
It also contains a [log browser]({{< relref "#log-browser" >}}) to further help you write queries.
For more information about Loki's query language, refer to the [Loki documentation](/docs/loki/latest/logql/).
### Use autocompletion
Code mode's autocompletion feature works automatically while typing.
The query editor can autocomplete static functions, aggregations, and keywords, and also dynamic items like labels.
The autocompletion dropdown includes documentation for the suggested items where available.
### Log browser
You can use the Loki log browser to navigate through your labels and values, and build queries.
**To navigate Loki and build a query:**
1. Choose labels to locate.
1. Search for the values of your selected labels.
The search field supports fuzzy search, and the log browser also supports faceting to list only possible label combinations.
1. Choose a query type between [**logs query**]({{< relref "#create-a-log-query" >}}) and [**rate metrics query**]({{< relref "#create-a-metric-query" >}}).
You can also validate the selector.
{{< figure src="/static/img/docs/v75/loki_log_browser.png" class="docs-image--no-shadow" max-width="800px" caption="The Loki log browser" >}}
### Configure query settings
| Name | Description |
| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Type** | Selects the query type to run. The `instant` type queries against a single point in time. We use the "To" time from the time range. The `range` type queries over the selected range of time. |
| **Line limit** | Defines the upper limit for the number of log lines returned by a query. The default is Loki's configured maximum lines limit. |
| **Legend** | _(Available only in a dashboard)_ Controls the time series name, using a name or pattern. For example, `{{hostname}}` is replaced with the label value for the label `hostname`. |
| **Resolution** | Sets the step parameter of Loki metrics range queries. With a resolution of `1/1`, each pixel corresponds to one data point. `1/2` retrieves one data point for every other pixel, `1/10` retrieves one data point per 10 pixels, and so on. Lower resolutions perform better. |
## Builder mode
Use Builder mode to visually construct queries, without needing to manually enter LogQL.
### Review toolbar features
In addition to the **Run query** button and mode switcher, Builder mode provides additional elements:
| Name | Description |
| ------------------------- | ----------------------------------------------------------------------------------------- |
| **Kick start your query** | A list of useful operation patterns you can use to add multiple operations to your query. |
| **Explain** | Displays a step-by-step explanation of all query components and operations. |
| **Raw query** | Displays the raw LogQL query that the Builder would send to Loki. |
### Use the Labels selector
Select labels and their values from the dropdown list.
When you select a label, Grafana retrieves available values from the server.
Use the `+` button to add a label and the `x` button to remove a label.
### Operations
Select the `+ Operations` button to add operations to your query.
The query editor groups operations into related sections, and you can type while the operations dropdown is open to search and filter the list.
The query editor displays a query's operations as boxes in the operations section.
Each operation's header displays its name, and additional action buttons appear when you hover your cursor over the header:
| Button | Action |
| ------ | ----------------------------------------------------------------- |
| `v` | Replaces the operation with different operation of the same type. |
| `info` | Opens the operation's description tooltip. |
| `x` | Removes the operation. |
Some operations have additional parameters under the operation header.
For details about each operation, use the `info` button to view the operation's description, or refer to the [Loki documentation](/docs/loki/latest/operations/).
Some operations make sense only when used in a specific order.
If adding an operation would result in nonsensical query, the query editor adds the operation to the correct place.
To re-order operations manually, drag the operation box by its name and drop it into the desired place.
#### Hints
In same cases the query editor can detect which operations would be most appropriate for a selected log stream. In such cases it will show a hint next to the `+ Operations` button. Click on the hint to add the operations to your query.
### Explain mode
Explain mode helps with understanding the query. It shows a step by step explanation of all query parts and the operations.
### Raw query
This section is shown only if the `Raw query` switch from the query editor top toolbar is set to `on`. It shows the raw query that will be created and executed by the query editor.
There are two types of LogQL queries:
- Log queries
- Metric queries
## Create a log query
Loki log queries return the contents of the log lines.
You can query and display log data from Loki via [Explore]({{< relref "../../../explore" >}}), and with the [Logs panel]({{< relref "../../../panels-visualizations/visualizations/logs" >}}) in dashboards.
To display the results of a log query, select the Loki data source, then enter a LogQL query.
For more information about log queries and LogQL, refer to the [Loki log queries documentation](/docs/loki/latest/logql/log_queries/).
### Show log context
When using a search expression as detailed above, you can retrieve the context surrounding your filtered results.
By clicking the `Show Context` link on the filtered rows, you'll be able to investigate the log messages that came before and after the
log message you're interested in.
### Tail live logs
Loki supports live tailing of logs in real-time in [Explore]({{< relref "../../../explore#loki-specific-features" >}}).
Live tailing relies on two Websocket connections: one between the browser and Grafana server, and another between the Grafana server and Loki server.
#### Proxying examples
If you use reverse proxies, configure them accordingly to use live tailing:
**Using Apache2 for proxying between the browser and the Grafana server:**
```
ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"
```
**Using NGINX:**
This example provides a basic NGINX proxy configuration.
It assumes that the Grafana server is available at `http://localhost:3000/`, the Loki server is running locally without proxy, and your external site uses HTTPS.
If you also host Loki behind an NGINX proxy, repeat the following configuration for Loki.
In the `http` section of NGINX configuration, add the following map definition:
```
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
```
In your `server` section, add the following configuration:
```
location ~ /(api/datasources/proxy/\d+/loki/api/v1/tail) {
proxy_pass http://localhost:3000$request_uri;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
proxy_set_header Connection $connection_upgrade;
proxy_set_header Upgrade $http_upgrade;
}
location / {
proxy_pass http://localhost:3000/;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto "https";
}
```
> **Note:** Available in Grafana v6.3 and higher.
## Create a metric query
You can use LogQL to wrap a log query with functions that create metrics from your logs.
For more information about metric queries, refer to the [Loki metric queries documentation](/docs/loki/latest/logql/metric_queries/).
## Apply annotations
[Annotations]({{< relref "../../../dashboards/build-dashboards/annotate-visualizations" >}}) overlay rich event information on top of graphs.
You can add annotation queries in the Dashboard menu's Annotations view.
You can use any non-metric Loki query as a source for annotations.
Grafana automatically uses log content as annotation text and your log stream labels as tags.
You don't need to create any additional mapping.
docs/sources/datasources/loki/template-variables/index.md
+49
View File
@@ -0,0 +1,49 @@
---
aliases:
- /docs/grafana/latest/data-sources/loki/template-variables/
description: Guide for using template variables when querying the Loki data source
keywords:
- grafana
- loki
- logs
- queries
- template
- variable
menuTitle: Template variables
title: Loki template variables
weight: 300
---
# Loki template variables
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
Grafana refers to such variables as template variables.
For an introduction to templating and template variables, refer to the [Templating]({{< relref "../../../dashboards/variables" >}}) and [Add and manage variables]({{< relref "../../../dashboards/variables/add-template-variables" >}}) documentation.
## Use query variables
Variables of the type _Query_ help you query Loki for lists of labels or label values.
The Loki data source provides a form to select the type of values expected for a given variable.
The form has these options:
| Query type | Example label | Example stream selector | List returned |
| ------------ | ------------- | ----------------------- | ---------------------------------------------------------------- |
| Label names | Not required | Not required | Label names. |
| Label values | `label` | | Label values for `label`. |
| Label values | `label` | `log stream selector` | Label values for `label` in the specified `log stream selector`. |
## Use ad hoc filters
Loki supports the special **Ad hoc filters** variable type.
You can use this variable type to specify any number of key/value filters, and Grafana applies them automatically to all of your Loki queries.
For more information, refer to [Add ad hoc filters]({{< relref "../../../dashboards/variables/add-template-variables#add-ad-hoc-filters" >}}).
## Use interval and range variables
You can use some global built-in variables in query variables: `$__interval`, `$__interval_ms`, `$__range`, `$__range_s`, and `$__range_ms`.
For more information, refer to [Global built-in variables]({{< relref "../../../dashboards/variables/add-template-variables#global-variables" >}}).