Docs: Refactor data sources content (#57573)
* Docs: Revise data source index * Docs: Consolidate data source administration docs * Docs: Revise panels docs related to data sources * Docs: Revise Alertmanager data source * Docs: Reorganize AWS CloudWatch data source docs * Docs: Reorganize Azure Monitor data source docs * Docs: Move azuremonitor to azure-monitor * Docs: Revise Elasticsearch docs * Docs: Move Elasticsearch index into bundle * Docs: Revise GCM docs * Docs: Revise Graphite docs * Docs: Move Graphite index into bundle * Docs: Revise InfluxDB docs * Docs: Revise Jaeger docs * Docs: Move Jaeger index into bundle * Docs: Revise Loki docs * Docs: Move Loki index into bundle * Docs: Revise MS SQL docs * Docs: Move MS SQL index into bundle * Docs: Revise Prometheus docs * Docs: Move Prometheus index into bundle * Docs: Revise Tempo docs * Docs: Move Tempo index into bundle * Docs: Revise TestData DB docs * Docs: Move TestData DB index into bundle * Docs: Revise Zipkin docs * Docs: Move Zipkin index into bundle * Docs: Move other data sources' index pages into bundles * Docs: Revise frontmatter * Fixing hugo markdown errors * Docs: Add query editor and template var sections to overview doc * Docs: Remove CTAs across data source docs * Docs: Remove CTA * Docs: Remove CTA * Docs: Fix links, images, typos, and usage consistency. * Docs: Fix typos * Docs: Fix CI issues * Update docs/sources/datasources/_index.md Co-authored-by: Torkel Ödegaard <torkel@grafana.com> * Update docs/sources/datasources/_index.md Co-authored-by: Torkel Ödegaard <torkel@grafana.com> * Docs: Fix query editor links * Update docs/sources/panels-visualizations/_index.md Co-authored-by: Torkel Ödegaard <torkel@grafana.com> * Update docs/sources/panels-visualizations/_index.md Co-authored-by: Torkel Ödegaard <torkel@grafana.com> * Docs: Rebundle child pages per writers' toolkit * Docs: Fix prettier for CI * Docs: Fix relrefs from outside data sources docs * Docs: Fix broken relrefs within datasources * Docs: Fix relrefs to data sources docs * Fixed some more refs Co-authored-by: Torkel Ödegaard <torkel@grafana.com>
This commit is contained in:
Garrett Guillotte
co-authored by
Torkel Ödegaard
Torkel Ödegaard
parent
22648d8581
commit
852d069a3c
@@ -0,0 +1,206 @@
|
||||
---
|
||||
aliases:
|
||||
- /docs/grafana/latest/datasources/elasticsearch/
|
||||
- /docs/grafana/latest/features/datasources/elasticsearch/
|
||||
- /docs/grafana/latest/data-sources/elasticsearch/
|
||||
description: Guide for using Elasticsearch in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- elasticsearch
|
||||
- guide
|
||||
menuTitle: Elasticsearch
|
||||
title: Elasticsearch data source
|
||||
weight: 325
|
||||
---
|
||||
|
||||
# Elasticsearch data source
|
||||
|
||||
Grafana ships with built-in support for Elasticsearch.
|
||||
You can make many types of queries to visualize logs or metrics stored in Elasticsearch, and annotate graphs with log events stored in Elasticsearch.
|
||||
|
||||
This topic explains configuring and querying specific to the Elasticsearch data source.
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data]({{< relref "../../panels-visualizations/query-transform-data" >}}).
|
||||
|
||||
For instructions on how to add a data source to Grafana, refer to the [administration documentation]({{< relref "../../administration/data-source-management/" >}}).
|
||||
Only users with the organization administrator role can add data sources.
|
||||
Administrators can also [configure the data source via YAML]({{< relref "#provision-the-data-source" >}}) with Grafana's provisioning system.
|
||||
|
||||
Once you've added the Elasticsearch data source, you can [configure it]({{< relref "#configure-the-data-source" >}}) so that your Grafana instance's users can create queries in its [query editor]({{< relref "./query-editor/" >}}) when they [build dashboards]({{< relref "../../dashboards/build-dashboards/" >}}) and use [Explore]({{< relref "../../explore" >}}).
|
||||
|
||||
## Supported Elasticsearch versions
|
||||
|
||||
This data source supports these versions of Elasticsearch:
|
||||
|
||||
- v7.10+
|
||||
- v8.0+ (experimental)
|
||||
|
||||
## Configure the data source
|
||||
|
||||
**To access the data source configuration page:**
|
||||
|
||||
1. Hover the cursor over the **Configuration** (gear) icon.
|
||||
1. Select **Data Sources**.
|
||||
1. Select the Elasticsearch data source.
|
||||
|
||||
Set the data source's basic configuration options carefully:
|
||||
|
||||
| Name | Description |
|
||||
| ----------- | -------------------------------------------------------------------------- |
|
||||
| **Name** | Sets the name you use to refer to the data source in panels and queries. |
|
||||
| **Default** | Sets the data source that's pre-selected for new panels. |
|
||||
| **Url** | Sets the HTTP protocol, IP, and port of your Elasticsearch server. |
|
||||
| **Access** | Don't modify Access. Use "Server (default)" or the data source won't work. |
|
||||
|
||||
You must also configure settings specific to the Elasticsearch data source.
|
||||
|
||||
### Index settings
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/elasticsearch-ds-details-7-4.png" max-width="500px" class="docs-image--right" caption="Elasticsearch data source details" >}}
|
||||
|
||||
Use the index settings to specify a default for the `time field` and your Elasticsearch index's name.
|
||||
You can use a time pattern, such as `YYYY.MM.DD`, or a wildcard for the index name.
|
||||
|
||||
### Elasticsearch version
|
||||
|
||||
Select the version of your Elasticsearch data source from the version selection dropdown.
|
||||
Different versions provide different query compositions and functionalities in the [query editor]({{< relref "./query-editor/" >}}).
|
||||
Available Elasticsearch versions are `2.x`, `5.x`, `5.6+`, `6.0+`, `7.0+`, `7.7+`, and `7.10+`.
|
||||
|
||||
Grafana assumes you're running the lowest possible version for a specified range.
|
||||
This ensures that new features or breaking changes in a future Elasticsearch release don't affect your configuration.
|
||||
|
||||
For example, if you run Elasticsearch `7.6.1` and select `7.0+`, and a new feature is made available for Elasticsearch `7.5.0` or newer releases, then a `7.5+` option will be available.
|
||||
However, your configuration won't be affected until you explicitly select the new `7.5+` option in your settings.
|
||||
|
||||
### Configure Min time interval
|
||||
|
||||
The **Min time interval** setting defines a lower limit for the auto group-by time interval.
|
||||
|
||||
This value _must_ be formatted as a number followed by a valid time identifier:
|
||||
|
||||
| Identifier | Description |
|
||||
| ---------- | ----------- |
|
||||
| `y` | year |
|
||||
| `M` | month |
|
||||
| `w` | week |
|
||||
| `d` | day |
|
||||
| `h` | hour |
|
||||
| `m` | minute |
|
||||
| `s` | second |
|
||||
| `ms` | millisecond |
|
||||
|
||||
We recommend setting this value to match your Elasticsearch write frequency.
|
||||
For example, set this to `1m` if Elasticsearch writes data every minute.
|
||||
|
||||
You can also override this setting in a dashboard panel under its data source options.
|
||||
|
||||
### X-Pack enabled
|
||||
|
||||
Toggle this to enable `X-Pack`-specific features and options, which provide the [query editor]({{< relref "./query-editor/" >}}) with additional aggregations, such as `Rate` and `Top Metrics`.
|
||||
|
||||
#### Include frozen indices
|
||||
|
||||
When the "X-Pack enabled" setting is active and the configured Elasticsearch version is higher than `6.6.0`, you can configure Grafana to not ignore [frozen indices](https://www.elastic.co/guide/en/elasticsearch/reference/7.13/frozen-indices.html) when performing search requests.
|
||||
|
||||
> **Note:** Frozen indices are [deprecated in Elasticsearch](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/frozen-indices.html) since v7.14.
|
||||
|
||||
### Logs
|
||||
|
||||
You can optionally configure the two Logs parameters **Message field name** and **Level field name** to determine which fields the data source uses for log messages and log levels when visualizing logs in [Explore]({{< relref "../../explore/" >}}).
|
||||
|
||||
For example, if you're using a default setup of Filebeat for shipping logs to Elasticsearch, set:
|
||||
|
||||
- **Message field name:** `message`
|
||||
- **Level field name:** `fields.level`
|
||||
|
||||
### Data links
|
||||
|
||||
Data links create a link from a specified field that can be accessed in Explore's logs view.
|
||||
|
||||
Each data link configuration consists of:
|
||||
|
||||
| Parameter | Description |
|
||||
| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| **Field** | Sets the name of the field used by the data link. |
|
||||
| **URL/query** | Sets the full link URL if the link is external. If the link is internal, this input serves as a query for the target data source.<br/>In both cases, you can interpolate the value from the field with the `${__value.raw }` macro. |
|
||||
| **URL Label** | (Optional) Sets a custom display label for the link. The link label defaults to the full external URL or name of the linked internal data source and is overridden by this setting. |
|
||||
| **Internal link** | Sets whether the link is internal or external. For an internal link, you can select the target data source with a data source selector. This supports only tracing data sources. |
|
||||
|
||||
### Configure Amazon Elasticsearch Service
|
||||
|
||||
If you use Amazon Elasticsearch Service, you can use Grafana's Elasticsearch data source to visualize data from it.
|
||||
|
||||
If you use an AWS Identity and Access Management (IAM) policy to control access to your Amazon Elasticsearch Service domain, you must use AWS Signature Version 4 (AWS SigV4) to sign all requests to that domain.
|
||||
|
||||
For details on AWS SigV4, refer to the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html).
|
||||
|
||||
#### AWS Signature Version 4 authentication
|
||||
|
||||
> **Note:** Available in Grafana v7.3 and higher.
|
||||
|
||||
To sign requests to your Amazon Elasticsearch Service domain, you can enable SigV4 in Grafana's [configuration]({{< relref "../../setup-grafana/configure-grafana/#sigv4_auth_enabled" >}}).
|
||||
|
||||
Once AWS SigV4 is enabled, you can configure it on the Elasticsearch data source configuration page.
|
||||
For more information about AWS authentication options, refer to [AWS authentication]({{< relref "../aws-cloudwatch/aws-authentication/" >}}).
|
||||
|
||||
{{< figure src="/static/img/docs/v73/elasticsearch-sigv4-config-editor.png" max-width="500px" class="docs-image--no-shadow" caption="SigV4 configuration for AWS Elasticsearch Service" >}}
|
||||
|
||||
### Provision the data source
|
||||
|
||||
You can define and configure the data source in YAML files as part of Grafana's provisioning system.
|
||||
For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana]({{< relref "../../administration/provisioning/#data-sources" >}}).
|
||||
|
||||
#### Provisioning examples
|
||||
|
||||
**Basic provisioning:**
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Elastic
|
||||
type: elasticsearch
|
||||
access: proxy
|
||||
database: '[metrics-]YYYY.MM.DD'
|
||||
url: http://localhost:9200
|
||||
jsonData:
|
||||
interval: Daily
|
||||
timeField: '@timestamp'
|
||||
```
|
||||
|
||||
**Provision for logs:**
|
||||
|
||||
```yaml
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: elasticsearch-v7-filebeat
|
||||
type: elasticsearch
|
||||
access: proxy
|
||||
database: '[filebeat-]YYYY.MM.DD'
|
||||
url: http://localhost:9200
|
||||
jsonData:
|
||||
interval: Daily
|
||||
timeField: '@timestamp'
|
||||
esVersion: '7.0.0'
|
||||
logMessageField: message
|
||||
logLevelField: fields.level
|
||||
dataLinks:
|
||||
- datasourceUid: my_jaeger_uid # Target UID needs to be known
|
||||
field: traceID
|
||||
url: '$${__value.raw}' # Careful about the double "$$" because of env var expansion
|
||||
```
|
||||
|
||||
## Query the data source
|
||||
|
||||
You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor.
|
||||
|
||||
For details, see the [query editor documentation]({{< relref "./query-editor/" >}}).
|
||||
|
||||
## Use template variables
|
||||
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
|
||||
Grafana refers to such variables as template variables.
|
||||
|
||||
For details, see the [template variables documentation]({{< relref "./template-variables/" >}}).
|
||||
@@ -0,0 +1,69 @@
|
||||
---
|
||||
aliases:
|
||||
- /docs/grafana/latest/data-sources/elasticsearch/query-editor/
|
||||
- /docs/grafana/latest/data-sources/elasticsearch/template-variables/
|
||||
description: Guide for using the Elasticsearch data source's query editor
|
||||
keywords:
|
||||
- grafana
|
||||
- elasticsearch
|
||||
- lucene
|
||||
- metrics
|
||||
- logs
|
||||
- queries
|
||||
menuTitle: Query editor
|
||||
title: Elasticsearch query editor
|
||||
weight: 300
|
||||
---
|
||||
|
||||
# Elasticsearch query editor
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/query-editor-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Elasticsearch Query Editor" >}}
|
||||
|
||||
This topic explains querying specific to the Elasticsearch data source.
|
||||
For general documentation on querying data sources in Grafana, see [Query and transform data]({{< relref "../../../panels-visualizations/query-transform-data" >}}).
|
||||
|
||||
## Select and edit metrics
|
||||
|
||||
You can select multiple metrics and group by multiple terms or filters when using the Elasticsearch query editor.
|
||||
|
||||
Use the plus and minus icons to the right to add and remove metrics or group by clauses.
|
||||
To expand the row to view and edit any available metric or group-by options, click the option text.
|
||||
|
||||
## Use template variables
|
||||
|
||||
You can also augment queries by using [template variables]({{< relref "./template-variables/" >}}).
|
||||
|
||||
## Name a time series
|
||||
|
||||
You can control the name for time series via the `Alias` input field.
|
||||
|
||||
| Pattern | Replacement value |
|
||||
| -------------------- | -------------------------------------- |
|
||||
| `{{term fieldname}}` | Value of a term group-by |
|
||||
| `{{metric}}` | Metric name, such as Average, Min, Max |
|
||||
| `{{field}}` | Metric field name |
|
||||
|
||||
## Control pipeline metrics visibility
|
||||
|
||||
Some metric aggregations, such as _Moving Average_ and _Derivative_, are called **Pipeline** aggregations.
|
||||
Elasticsearch pipeline metrics must be based on another metric.
|
||||
|
||||
Use the eye icon next to the metric to prevent metrics from appearing in the graph.
|
||||
This is useful for metrics you only have in the query for use in a pipeline metric.
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/pipeline-aggregation-editor-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Pipeline aggregation editor" >}}
|
||||
|
||||
## Create a query
|
||||
|
||||
Write the query using a custom JSON string, with the field mapped as a [keyword](https://www.elastic.co/guide/en/elasticsearch/reference/current/keyword.html#keyword) in the Elasticsearch index mapping.
|
||||
|
||||
If the query is [multi-field](https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html) with both a `text` and `keyword` type, use `"field":"fieldname.keyword"` (sometimes `fieldname.raw`) to specify the keyword field in your query.
|
||||
|
||||
| Query | Description |
|
||||
| ------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `{"find": "fields", "type": "keyword"}` | Returns a list of field names with the index type `keyword`. |
|
||||
| `{"find": "terms", "field": "hostname.keyword", "size": 1000}` | Returns a list of values for a keyword using term aggregation. Query will use current dashboard time range as time range query. |
|
||||
| `{"find": "terms", "field": "hostname", "query": '<Lucene query>'}` | Returns a list of values for a keyword field using term aggregation and a specified Lucene query filter. Query will use current dashboard time range as time range for query. |
|
||||
|
||||
Queries of `terms` have a 500-result limit by default.
|
||||
To set a custom limit, set the `size` property in your query.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
aliases:
|
||||
- /docs/grafana/latest/datasources/elasticsearch/template-variables/
|
||||
- /docs/grafana/latest/data-sources/elasticsearch/template-variables/
|
||||
description: Using template variables with Elasticsearch in Grafana
|
||||
keywords:
|
||||
- grafana
|
||||
- elasticsearch
|
||||
- templates
|
||||
- variables
|
||||
- queries
|
||||
menuTitle: Template variables
|
||||
title: Elasticsearch template variables
|
||||
weight: 400
|
||||
---
|
||||
|
||||
# Elasticsearch template variables
|
||||
|
||||
Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables.
|
||||
Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard.
|
||||
Grafana refers to such variables as template variables.
|
||||
|
||||
For an introduction to templating and template variables, refer to the [Templating]({{< relref "../../../dashboards/variables" >}}) and [Add and manage variables]({{< relref "../../../dashboards/variables/add-template-variables" >}}) documentation.
|
||||
|
||||
## Choose a variable syntax
|
||||
|
||||
The Elasticsearch data source supports two variable syntaxes for use in the **Query** field:
|
||||
|
||||
- `$varname`, such as `hostname:$hostname`, which is easy to read and write but doesn't let you use a variable in the middle of a word.
|
||||
- `[[varname]]`, such as `hostname:[[hostname]]`
|
||||
|
||||
When the _Multi-value_ or _Include all value_ options are enabled, Grafana converts the labels from plain text to a Lucene-compatible condition.
|
||||
For details, see the [Multi-value variables]({{< relref "../../../dashboards/variables/add-template-variables#multi-value-variables" >}}) documentation.
|
||||
|
||||
## Use variables in queries
|
||||
|
||||
You can use other variables inside the query.
|
||||
This example is used to define a variable named `$host`:
|
||||
|
||||
```
|
||||
{"find": "terms", "field": "hostname", "query": "source:$source"}
|
||||
```
|
||||
|
||||
This uses another variable named `$source` inside the query definition.
|
||||
Whenever you change the value of the `$source` variable via the dropdown, Grafana triggers an update of the `$host` variable to contain only hostnames filtered by, in this case, the `source` document property.
|
||||
|
||||
These queries by default return results in term order (which can then be sorted alphabetically or numerically as for any variable).
|
||||
To produce a list of terms sorted by doc count (a top-N values list), add an `orderBy` property of "doc_count".
|
||||
This automatically selects a descending sort.
|
||||
|
||||
> **Note:** To use an ascending sort (`asc`) with doc_count (a bottom-N list), set `order: "asc"`. However, Elasticsearch [discourages this](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order) because sorting by ascending doc count can return inaccurate results.
|
||||
|
||||
To keep terms in the doc count order, set the variable's Sort dropdown to **Disabled**.
|
||||
You can alternatively use other sorting criteria, such as **Alphabetical**, to re-sort them.
|
||||
|
||||
```
|
||||
{"find": "terms", "field": "hostname", "orderBy": "doc_count"}
|
||||
```
|
||||
|
||||
## Template variable examples
|
||||
|
||||
{{< figure src="/static/img/docs/elasticsearch/elastic-templating-query-7-4.png" max-width="500px" class="docs-image--no-shadow" caption="Query with template variables" >}}
|
||||
|
||||
In the above example, a Lucene query filters documents based on the `hostname` property using a variable named `$hostname`.
|
||||
The example also uses a variable in the _Terms_ group by field input box, which you can use to quickly change how data is grouped.
|
||||
|
||||
To view an example dashboard on Grafana Play, see the [Elasticsearch Templated Dashboard](https://play.grafana.org/d/CknOEXDMk/elasticsearch-templated?orgId=1d).
|
||||
Reference in New Issue
Block a user