public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Docs: Move files (#27983)

Browse Source 
* moved datasources folder

* Update menu.yaml

* updated Azuremonitor

* Update azuremonitor.md

* Update cloudmonitoring.md

* Update cloudwatch.md

* updated content

* Update elasticsearch.md

* Update graphite.md

* Update elasticsearch.md

* Update azuremonitor.md

* Update cloudmonitoring.md

* Update influxdb.md

* Update jaeger.md

* Update loki.md

* Update mssql.md

* Update mysql.md

* Update opentsdb.md

* Update postgres.md

* Update prometheus.md

* Update postgres.md

* Update jaeger.md

* Update mysql.md

* Update mssql.md

* added aliases

* Update testdata.md

* Update zipkin.md

* link fixes

* link fixing

* moved files out of guides

* Delete archive.md

* updated links

* updated aliases

* updated aliases

* link updates

* Update _index.md

* deleted archive.md

* Update azuremonitor.md

* Update azuremonitor.md
This commit is contained in:
Diana Payton
2020-10-09 08:16:25 -07:00
committed by GitHub
parent 873c32bc97
commit 90303b5a03
65 changed files with 257 additions and 297 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sources/datasources/_index.md
+49
View File
@@ -0,0 +1,49 @@
+++
title = "Data sources"
type = "docs"
aliases = ["/docs/grafana/latest/datasources/overview/"]
[menu.docs]
name = "Data sources"
identifier = "datasources"
parent = "features"
weight = 50
+++
# Data sources
Grafana supports many different storage backends for your time series data (data source). Each data source has a specific Query Editor that is customized for the features and capabilities that the particular data source exposes.
## Querying
The query language and capabilities of each data source are obviously very different. You can combine data from multiple data sources onto a single Dashboard, but each Panel is tied to a specific data source that belongs to a particular Organization.
## Supported data sources
The following data sources are officially supported:
- [AWS CloudWatch]({{< relref "cloudwatch.md" >}})
- [Azure Monitor]({{< relref "azuremonitor.md" >}})
- [Elasticsearch]({{< relref "elasticsearch.md" >}})
- [Google Cloud Monitoring]({{< relref "cloudmonitoring.md" >}})
- [Graphite]({{< relref "graphite.md" >}})
- [InfluxDB]({{< relref "influxdb.md" >}})
- [Loki]({{< relref "loki.md" >}})
- [Microsoft SQL Server (MSSQL)]({{< relref "mssql.md" >}})
- [MySQL]({{< relref "mysql.md" >}})
- [OpenTSDB]({{< relref "opentsdb.md" >}})
- [PostgreSQL]({{< relref "postgres.md" >}})
- [Prometheus]({{< relref "prometheus.md" >}})
- [Testdata]({{< relref "testdata.md" >}})
In addition to the data sources that you have configured in your Grafana, there are three special data sources available:
- **Grafana -** A built-in data source that generates random walk data. Useful for testing visualizations and running experiments.
- **Mixed -** Select this to query multiple data sources in the same panel. When this data source is selected, Grafana allows you to select a data source for every new query that you add.
- The first query will use the data source that was selected before you selected **Mixed**.
- You cannot change an existing query to use the Mixed Data Source.
- Grafana Play example: [Mixed data sources](https://play.grafana.org/d/000000100/mixed-datasources?orgId=1)
- **Dashboard -** Select this to use a result set from another panel in the same dashboard.
## Data source plugins
Since Grafana 3.0 you can install data sources as plugins. Check out [Grafana.com/plugins](https://grafana.com/plugins) for more data sources.
docs/sources/datasources/add-a-data-source.md
+32
View File
@@ -0,0 +1,32 @@
+++
title = "Add data source"
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/add-a-data-source/"]
[menu.docs]
name = "Add a data source"
identifier = "add_data_source"
parent = "features"
weight = 100
+++
# Add a data source
Before you create your first dashboard, you need to add your data source. Following are the list of instructions to create one.
> **Note:** Only users with the organization Admin role can add data sources.
1. Move your cursor to the cog on the side menu which will show you the configuration menu. If the side menu is not visible click the Grafana icon in the upper left corner. Click on **Configuration** > **Data Sources** in the side menu and you'll be taken to the data sources page
where you can add and edit data sources. You can also click the cog.
{{< docs-imagebox img="/img/docs/v52/sidemenu-datasource.png" max-width="250px" class="docs-image--no-shadow">}}
1. Click **Add data source** and you will come to the settings page of your new data source.
{{< docs-imagebox img="/img/docs/v52/add-datasource.png" max-width="700px" class="docs-image--no-shadow">}}
1. In the **Name** box, enter a name for this data source.
{{< docs-imagebox img="/img/docs/v52/datasource-settings.png" max-width="700px" class="docs-image--no-shadow">}}
1. In the **Type**, select the type of data source. See [Supported data sources]({{< relref "_index.md#supported-data-sources" >}}) for more information and how to configure your data source settings.
1. Click **Save & Test**.
docs/sources/datasources/azuremonitor.md
+413
View File
@@ -0,0 +1,413 @@
+++
title = "Azure Monitor"
description = "Guide for using Azure Monitor in Grafana"
keywords = ["grafana", "microsoft", "azure", "monitor", "application", "insights", "log", "analytics", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/azuremonitor"]
[menu.docs]
name = "Azure Monitor"
parent = "datasources"
weight = 300
+++
# Using Azure Monitor in Grafana
> Officially released in Grafana v6.0.0
As of Grafana 6.0, the Azure Monitor plugin has been moved into Grafana so it now ships with built-in support for Azure Monitor.
The Azure Monitor data source supports multiple services in the Azure cloud:
- **[Azure Monitor]({{< relref "#querying-the-azure-monitor-service" >}})** is the platform service that provides a single source for monitoring Azure resources.
- **[Application Insights]({{< relref "#querying-the-application-insights-service" >}})** is an extensible Application Performance Management (APM) service for web developers on multiple platforms and can be used to monitor your live web application - it will automatically detect performance anomalies.
- **[Azure Log Analytics]({{< relref "#querying-the-azure-log-analytics-service" >}})** (or Azure Logs) gives you access to log data collected by Azure Monitor.
- **[Application Insights Analytics]({{< relref "#query-the-application-insights-analytics-service" >}})** allows you to query [Application Insights data](https://docs.microsoft.com/en-us/azure/azure-monitor/app/analytics) using the same query language used for Azure Log Analytics.
## Add the data source
The data source can access metrics from four different services. You can configure access to the services that you use. It is also possible to use the same credentials for multiple services if that is how you have set it up in Azure AD.
- [Guide to setting up an Azure Active Directory Application for Azure Monitor.](https://docs.microsoft.com/en-us/azure/azure-resource-manager/resource-group-create-service-principal-portal)
- [Guide to setting up an Azure Active Directory Application for Azure Log Analytics.](https://dev.loganalytics.io/documentation/Authorization/AAD-Setup)
- [Quickstart Guide for Application Insights.](https://dev.applicationinsights.io/quickstart/)
1. Accessed from the Grafana main menu, newly installed data sources can be added immediately within the Data Sources section. Next, click the "Add data source" button in the upper right. The Azure Monitor data source will be available for selection in the Cloud section in the list of data sources.
1. In the name field, Grafana will automatically fill in a name for the data source - `Azure Monitor` or something like `Azure Monitor - 3`. If you are going to configure multiple data sources, then change the name to something more informative.
1. If you are using Azure Monitor, you need 4 pieces of information from the Azure portal (see link above for detailed instructions):
- **Tenant Id** (Azure Active Directory -> Properties -> Directory ID)
- **Client Id** (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
- **Client Secret** (Azure Active Directory -> App Registrations -> Choose your app -> Keys)
- **Default Subscription Id** (Subscriptions -> Choose subscription -> Overview -> Subscription ID)
1. Paste these four items into the fields in the Azure Monitor API Details section:
{{< docs-imagebox img="/img/docs/v62/config_1_azure_monitor_details.png" class="docs-image--no-shadow" caption="Azure Monitor Configuration Details" >}}
- The Subscription Id can be changed per query. Save the data source and refresh the page to see the list of subscriptions available for the specified Client Id.
1. If you are also using the Azure Log Analytics service, then you need to specify these two config values (or you can reuse the Client Id and Secret from the previous step).
- Client Id (Azure Active Directory -> App Registrations -> Choose your app -> Application ID)
- Client Secret (Azure Active Directory -> App Registrations -> Choose your app -> Keys -> Create a key -> Use client secret)
1. If you are using Application Insights, then you need two pieces of information from the Azure Portal (see link above for detailed instructions):
- Application ID
- API Key
1. Paste these two items into the appropriate fields in the Application Insights API Details section:
{{< docs-imagebox img="/img/docs/v62/config_2_app_insights_api_details.png" class="docs-image--no-shadow" caption="Application Insights Configuration Details" >}}
1. Test that the configuration details are correct by clicking on the "Save & Test" button:
{{< docs-imagebox img="/img/docs/v62/config_3_save_and_test.png" class="docs-image--no-shadow" caption="Save and Test" >}}
Alternatively on step 4 if creating a new Azure Active Directory App, use the [Azure CLI](https://docs.microsoft.com/en-us/cli/azure/?view=azure-cli-latest):
```bash
az ad sp create-for-rbac -n "http://localhost:3000"
```
## Choose a Service
In the query editor for a panel, after choosing your Azure Monitor data source, the first option is to choose a service. There are three options here:
- `Azure Monitor`
- `Application Insights`
- `Azure Log Analytics`
- `Insights Analytics`
The query editor changes depending on which one you pick. Azure Monitor is the default.
Starting in Grafana 7.1, Insights Analytics replaced the former edit mode from within Application Insights.
## Query the Azure Monitor service
The Azure Monitor service provides metrics for all the Azure services that you have running. It helps you understand how your applications on Azure are performing and to proactively find issues affecting your applications.
If your Azure Monitor credentials give you access to multiple subscriptions, then choose the appropriate subscription first.
Examples of metrics that you can get from the service are:
- `Microsoft.Compute/virtualMachines - Percentage CPU`
- `Microsoft.Network/networkInterfaces - Bytes sent`
- `Microsoft.Storage/storageAccounts - Used Capacity`
{{< docs-imagebox img="/img/docs/v60/azuremonitor-service-query-editor.png" class="docs-image--no-shadow" caption="Azure Monitor Query Editor" >}}
As of Grafana 7.1, the query editor allows you to query multiple dimensions for metrics that support them. Metrics that support multiple dimensions are those listed in the [Azure Monitor supported Metrics List](https://docs.microsoft.com/en-us/azure/azure-monitor/platform/metrics-supported) that have one or more values listed in the "Dimension" column for the metric.
### Format legend keys with aliases for Azure Monitor
The default legend formatting for the Azure Monitor API is:
`metricName{dimensionName=dimensionValue,dimensionTwoName=DimensionTwoValue}`
> **Note:** Before Grafana 7.1, the formatting included the resource name in the default: `resourceName{dimensionName=dimensionValue}.metricName`. As of Grafana 7.1, the resource name has been removed from the default legend.
These can be quite long, but this formatting can be changed by using aliases. In the **Legend Format** field, you can combine the aliases defined below any way you want.
Azure Monitor examples:
- `Blob Type: {{ blobtype }}`
- `{{ resourcegroup }} - {{ resourcename }}`
### Alias patterns for Azure Monitor
- `{{ resourcegroup }}` = replaced with the value of the Resource Group
- `{{ namespace }}` = replaced with the value of the Namespace (e.g. Microsoft.Compute/virtualMachines)
- `{{ resourcename }}` = replaced with the value of the Resource Name
- `{{ metric }}` = replaced with metric name (e.g. Percentage CPU)
- `{{ dimensionname }}` = _Legacy as of 7.1+ (for backwards compatibility)_ replaced with the first dimension's key/label (as sorted by the key/label) (e.g. blobtype)
- `{{ dimensionvalue }}` = _Legacy as of 7.1+ (for backwards compatibility)_ replaced with first dimension's value (as sorted by the key/label) (e.g. BlockBlob)
- `{{ arbitraryDim }}` = _Available in 7.1+_ replaced with the value of the corresponding dimension. (e.g. `{{ blobtype }}` becomes BlockBlob)
### Create template variables for Azure Monitor
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.
Note that the Azure Monitor service does not support multiple values yet. If you want to visualize multiple time series (for example, metrics for server1 and server2) then you have to add multiple queries to able to view them on the same graph or in the same table.
The Azure Monitor data source Plugin provides the following queries you can specify in the `Query` field in the Variable edit view. They allow you to fill a variable's options list.
| Name | Description |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------- |
| _Subscriptions()_ | Returns a list of subscriptions. |
| _ResourceGroups()_ | Returns a list of resource groups. |
| _ResourceGroups(12345678-aaaa-bbbb-cccc-123456789aaa)_ | Returns a list of resource groups for a specified subscription. |
| _Namespaces(aResourceGroup)_ | Returns a list of namespaces for the specified resource group. |
| _Namespaces(12345678-aaaa-bbbb-cccc-123456789aaa, aResourceGroup)_ | Returns a list of namespaces for the specified resource group and subscription. |
| _ResourceNames(aResourceGroup, aNamespace)_ | Returns a list of resource names. |
| _ResourceNames(12345678-aaaa-bbbb-cccc-123456789aaa, aResourceGroup, aNamespace)_ | Returns a list of resource names for a specified subscription. |
| _MetricNamespace(aResourceGroup, aNamespace, aResourceName)_ | Returns a list of metric namespaces. |
| _MetricNamespace(12345678-aaaa-bbbb-cccc-123456789aaa, aResourceGroup, aNamespace, aResourceName)_ | Returns a list of metric namespaces for a specified subscription. |
| _MetricNames(aResourceGroup, aNamespace, aResourceName)_ | Returns a list of metric names. |
| _MetricNames(12345678-aaaa-bbbb-cccc-123456789aaa, aResourceGroup, aNamespace, aResourceName)_ | Returns a list of metric names for a specified subscription. |
Examples:
- Resource Groups query: `ResourceGroups()`
- Passing in metric name variable: `Namespaces(cosmo)`
- Chaining template variables: `ResourceNames($rg, $ns)`
- Do not quote parameters: `MetricNames(hg, Microsoft.Network/publicIPAddresses, grafanaIP)`
{{< docs-imagebox img="/img/docs/v60/azuremonitor-service-variables.png" class="docs-image--no-shadow" caption="Nested Azure Monitor Template Variables" >}}
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### List of supported Azure Monitor metrics
Not all metrics returned by the Azure Monitor API have values. To make it easier for you when building a query, the Grafana data source has a list of supported Azure Monitor metrics and ignores metrics which will never have values. This list is updated regularly as new services and metrics are added to the Azure cloud. For more information about the list of Azure Monitor metrics, refer to [current supported namespaces](https://github.com/grafana/grafana/blob/master/public/app/plugins/datasource/grafana-azure-monitor-datasource/azure_monitor/supported_namespaces.ts).
### Azure Monitor alerting
Grafana alerting is supported for the Azure Monitor service. This is not Azure Alerts support. For more information about Grafana alerting, refer to [how alerting in Grafana works]({{< relref "../alerting/alerts-overview.md" >}}).
{{< docs-imagebox img="/img/docs/v60/azuremonitor-alerting.png" class="docs-image--no-shadow" caption="Azure Monitor Alerting" >}}
## Query the Application Insights Service
{{< docs-imagebox img="/img/docs/azuremonitor/insights_metrics_multi-dim.png" class="docs-image--no-shadow" caption="Application Insights Query Editor" >}}
As of Grafana 7.1, you can select more than one group by dimension.
### Formatting legend keys with aliases for Application Insights
The default legend formatting is:
`metricName{dimensionName=dimensionValue,dimensionTwoName=DimensionTwoValue}`
In the Legend Format field, the aliases which are defined below can be combined any way you want.
Application Insights examples:
- `city: {{ client/city }}`
- `{{ metric }} [Location: {{ client/countryOrRegion }}, {{ client/city }}]`
### Alias patterns for Application Insights
- `{{ groupbyvalue }}` = _Legacy as of 7.1+ (for backwards compatibility)_ replaced with the first dimension's key/label (as sorted by the key/label)
- `{{ groupbyname }}` = _Legacy as of 7.1+ (for backwards compatibility)_ replaced with first dimension's value (as sorted by the key/label) (e.g. BlockBlob)
- `{{ metric }}` = replaced with metric name (e.g. requests/count)
- `{{ arbitraryDim }}` = _Available in 7.1+_ replaced with the value of the corresponding dimension. (e.g. `{{ client/city }}` becomes Chicago)
### Filter expressions for Application Insights
The filter field takes an OData filter expression.
Examples:
- `client/city eq 'Boydton'`
- `client/city ne 'Boydton'`
- `client/city ne 'Boydton' and client/city ne 'Dublin'`
- `client/city eq 'Boydton' or client/city eq 'Dublin'`
### Templating with variables for Application Insights
Use the one of the following queries in the `Query` field in the Variable edit view.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
| Name | Description |
| ---------------------------------- | ------------------------------------------------------------ |
| _AppInsightsMetricNames()_ | Returns a list of metric names. |
| _AppInsightsGroupBys(aMetricName)_ | Returns a list of "group bys" for the specified metric name. |
Examples:
- Metric Names query: `AppInsightsMetricNames()`
- Passing in metric name variable: `AppInsightsGroupBys(requests/count)`
- Chaining template variables: `AppInsightsGroupBys($metricnames)`
{{< docs-imagebox img="/img/docs/v60/appinsights-service-variables.png" class="docs-image--no-shadow" caption="Nested Application Insights Template Variables" >}}
### Application Insights alerting
Grafana alerting is supported for Application Insights. This is not Azure Alerts support. For more information about Grafana alerting, refer to [Alerts overview]({{< relref "../alerting/alerts-overview.md" >}}).
{{< docs-imagebox img="/img/docs/v60/azuremonitor-alerting.png" class="docs-image--no-shadow" caption="Azure Monitor Alerting" >}}
## Querying the Azure Log Analytics service
Queries are written in the new [Azure Log Analytics (or KustoDB) Query Language](https://docs.microsoft.com/en-us/azure/azure-monitor/log-query/query-language). A Log Analytics query can be formatted as time series data or as table data.
If your credentials give you access to multiple subscriptions, then choose the appropriate subscription before entering queries.
### Time series queries
Time series queries are for the Graph panel and other panels like the SingleStat panel. Each query must contain at least a datetime column and a numeric value column. The result must also be sorted in ascending order by the datetime column.
Here is an example query that returns the aggregated count grouped by hour:
```kusto
Perf
| where $__timeFilter(TimeGenerated)
| summarize count() by bin(TimeGenerated, 1h)
| order by TimeGenerated asc
```
A query can also have one or more non-numeric/non-datetime columns, and those columns are considered dimensions and become labels in the response. For example, a query that returns the aggregated count grouped by hour, Computer, and the CounterName:
```kusto
Perf
| where $__timeFilter(TimeGenerated)
| summarize count() by bin(TimeGenerated, 1h), Computer, CounterName
| order by TimeGenerated asc
```
You can also select additional number value columns (with, or without multiple dimensions). For example, getting a count and average value by hour, Computer, CounterName, and InstanceName:
```kusto
Perf
| where $__timeFilter(TimeGenerated)
| summarize Samples=count(), ["Avg Value"]=avg(CounterValue)
by bin(TimeGenerated, $__interval), Computer, CounterName, InstanceName
| order by TimeGenerated asc
```
> **Tip**: In the above query, the Kusto syntax `Samples=count()` and `["Avg Value"]=...` is used to rename those columns — the second syntax allowing for the space. This changes the name of the metric that Grafana uses, and as a result, things like series legends and table columns will match what you specify. Here `Samples` is displayed instead of `_count`.
{{< docs-imagebox img="/img/docs/azuremonitor/logs_multi-value_multi-dim.png" class="docs-image--no-shadow" caption="Azure Logs query with multiple values and multiple dimensions" >}}
### Table queries
Table queries are mainly used in the Table panel and show a list of columns and rows. This example query returns rows with the six specified columns:
```kusto
AzureActivity
| where $__timeFilter()
| project TimeGenerated, ResourceGroup, Category, OperationName, ActivityStatus, Caller
| order by TimeGenerated desc
```
### Format the display name for Log Analytics
The default display name format is:
`metricName{dimensionName=dimensionValue,dimensionTwoName=DimensionTwoValue}`
This can be customized by using the [display name field option]({{< relref "../panels/field-options/_index.md#display-name" >}}).
### Azure Log Analytics macros
To make writing queries easier there are several Grafana macros that can be used in the where clause of a query:
- `$__timeFilter()` - Expands to
`TimeGenerated ≥ datetime(2018-06-05T18:09:58.907Z) and`
`TimeGenerated ≤ datetime(2018-06-05T20:09:58.907Z)` where the from and to datetimes are from the Grafana time picker.
- `$__timeFilter(datetimeColumn)` - Expands to
`datetimeColumn ≥ datetime(2018-06-05T18:09:58.907Z) and`
`datetimeColumn ≤ datetime(2018-06-05T20:09:58.907Z)` where the from and to datetimes are from the Grafana time picker.
- `$__timeFrom()` - Returns the From datetime from the Grafana picker. Example: `datetime(2018-06-05T18:09:58.907Z)`.
- `$__timeTo()` - Returns the From datetime from the Grafana picker. Example: `datetime(2018-06-05T20:09:58.907Z)`.
- `$__escapeMulti($myVar)` - is to be used with multi-value template variables that contain illegal characters. If `$myVar` has the following two values as a string `'\\grafana-vm\Network(eth0)\Total','\\hello!'`, then it expands to: `@'\\grafana-vm\Network(eth0)\Total', @'\\hello!'`. If using single value variables there is no need for this macro, simply escape the variable inline instead - `@'\$myVar'`.
- `$__contains(colName, $myVar)` - is to be used with multi-value template variables. If `$myVar` has the value `'value1','value2'`, it expands to: `colName in ('value1','value2')`.
If using the `All` option, then check the `Include All Option` checkbox and in the `Custom all value` field type in the following value: `all`. If `$myVar` has value `all` then the macro will instead expand to `1 == 1`. For template variables with a lot of options, this will increase the query performance by not building a large "where..in" clause.
### Azure Log Analytics builtin variables
There are also some Grafana variables that can be used in Azure Log Analytics queries:
- `$__interval` - Grafana calculates the minimum time grain that can be used to group by time in queries. For more information about `$__interval`, refer to [interval variables]({{< relref "../variables/variable-types/_index.md#interval-variables" >}}). It returns a time grain like `5m` or `1h` that can be used in the bin function. E.g. `summarize count() by bin(TimeGenerated, $__interval)`
### Templating with variables for Azure Log Analytics
Any Log Analytics query that returns a list of values can be used in the `Query` field in the Variable edit view. There is also one Grafana function for Log Analytics that returns a list of workspaces.
Refer to the [Variables]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
| Name | Description |
| -------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| _workspaces()_ | Returns a list of workspaces for the default subscription. |
| _workspaces(12345678-aaaa-bbbb-cccc-123456789aaa)_ | Returns a list of workspaces for the specified subscription (the parameter can be quoted or unquoted). |
Example variable queries:
<!-- prettier-ignore-start -->
| Query | Description |
| --------------------------------------------------------------------------------------- | --------------------------------------------------------- |
| _subscriptions()_ | Returns a list of Azure subscriptions |
| _workspaces()_ | Returns a list of workspaces for default subscription |
| _workspaces("12345678-aaaa-bbbb-cccc-123456789aaa")_ | Returns a list of workspaces for a specified subscription |
| _workspaces("$subscription")_ | With template variable for the subscription parameter |
| _workspace("myWorkspace").Heartbeat \| distinct Computer_ | Returns a list of Virtual Machines |
| _workspace("$workspace").Heartbeat \| distinct Computer_ | Returns a list of Virtual Machines with template variable |
| _workspace("$workspace").Perf \| distinct ObjectName_ | Returns a list of objects from the Perf table |
| _workspace("$workspace").Perf \| where ObjectName == "$object" \| distinct CounterName_ | Returns a list of metric names from the Perf table |
<!-- prettier-ignore-end -->
Example of a time series query using variables:
```kusto
Perf
| where ObjectName == "$object" and CounterName == "$metric"
| where TimeGenerated >= $__timeFrom() and TimeGenerated <= $__timeTo()
| where $__contains(Computer, $computer)
| summarize avg(CounterValue) by bin(TimeGenerated, $__interval), Computer
| order by TimeGenerated asc
```
### Deep linking from Grafana panels to the Log Analytics query editor in Azure Portal
> Only available in Grafana v7.0+.
{{< docs-imagebox img="/img/docs/v70/azure-log-analytics-deep-linking.png" max-width="500px" class="docs-image--right" caption="Azure Log Analytics deep linking" >}}
Click on a time series in the panel to see a context menu with a link to `View in Azure Portal`. Clicking that link opens the Azure Log Analytics query editor in the Azure Portal and runs the query from the Grafana panel there.
If you're not currently logged in to the Azure Portal, then the link opens the login page. The provided link is valid for any account, but it only displays the query if your account has access to the Azure Log Analytics workspace specified in the query.
<div class="clearfix"></div>
### Azure Log Analytics alerting
> Only available in Grafana v7.0+.
Grafana alerting is supported for Application Insights. This is not Azure Alerts support. Read more about how alerting in Grafana works in [Alerting rules]({{< relref "../alerting/alerts-overview.md" >}}).
## Query the Application Insights Analytics service
If you change the service type to **Insights Analytics**, then a similar editor to the Log Analytics service is available. This service also uses the Kusto language, so the instructions for querying data are identical to [querying the log analytics service]({{< relref "#querying-the-azure-log-analytics-service" >}}), except that you query Application Insights Analytics data instead.
{{< docs-imagebox img="/img/docs/azuremonitor/insights_analytics_multi-dim.png" class="docs-image--no-shadow" caption="Azure Application Insights Analytics query with multiple dimensions" >}}
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
# config file version
apiVersion: 1
datasources:
- name: Azure Monitor
type: grafana-azure-monitor-datasource
access: proxy
jsonData:
appInsightsAppId: <app-insights-app-id>
clientId: <client-id>
cloudName: azuremonitor
subscriptionId: <subscription-id>
tenantId: <tenant-id>
logAnalyticsClientId: <log-analytics-client-id>
logAnalyticsDefaultWorkspace: <log-analytics-default-workspace>
logAnalyticsSubscriptionId: <log-analytics-subscription-id>
logAnalyticsTenantId: <log-analytics-tenant-id>
secureJsonData:
clientSecret: <client-secret>
appInsightsApiKey: <app-insights-api-key>
logAnalyticsClientSecret: <log-analytics-client-secret>
version: 1
```
docs/sources/datasources/cloudmonitoring.md
+337
View File
@@ -0,0 +1,337 @@
+++
title = "Cloud Monitoring"
description = "Guide for using Google Cloud Monitoring in Grafana"
keywords = ["grafana", "stackdriver", "google", "guide", "cloud", "monitoring"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/stackdriver", "/docs/grafana/latest/features/datasources/cloudmonitoring/"]
[menu.docs]
name = "Google Cloud Monitoring"
parent = "datasources"
weight = 200
+++
# Using Google Cloud Monitoring in Grafana
> Officially released in Grafana v6.0.0
> Before Grafana v7.1 this data source was named Google Stackdriver.
Grafana ships with built-in support for Google Cloud Monitoring. Just add it as a data source and you are ready to build dashboards for your Google Cloud Monitoring metrics.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Dashboards` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select `Google Cloud Monitoring` from the _Type_ dropdown.
1. Upload or paste in the Service Account Key file. See below for steps on how to create a Service Account Key file.
> **Note:** If you're not seeing the `Data Sources` link in your side menu, then your current user account does not have the `Admin` role for the current organization.
| Name | Description |
| --------------------- | ------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels and queries. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _Service Account Key_ | Service Account Key File for a GCP Project. Instructions below on how to create it. |
## Authentication
There are two ways to authenticate the Google Cloud Monitoring plugin - either by uploading a Google JWT file, or by automatically retrieving credentials from Google metadata server. The latter option is only available when running Grafana on GCE virtual machine.
### Using a Google Service Account Key File
To authenticate with the Google Cloud Monitoring API, you need to create a Google Cloud Platform (GCP) Service Account for the Project you want to show data for. A Grafana data source integrates with one GCP Project. If you want to visualize data from multiple GCP Projects then you need to create one data source per GCP Project.
#### Enable APIs
The following APIs need to be enabled first:
- [Monitoring API](https://console.cloud.google.com/apis/library/monitoring.googleapis.com)
- [Cloud Resource Manager API](https://console.cloud.google.com/apis/library/cloudresourcemanager.googleapis.com)
Click on the links above and click the `Enable` button:
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_enable_api.png" class="docs-image--no-shadow" caption="Enable GCP APIs" >}}
#### Create a GCP Service Account for a Project
1. Navigate to the [APIs and Services Credentials page](https://console.cloud.google.com/apis/credentials).
1. Click on the `Create credentials` dropdown/button and choose the `Service account key` option.
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_create_service_account_button.png" class="docs-image--no-shadow" caption="Create service account button" >}}
1. On the `Create service account key` page, choose key type `JSON`. Then in the `Service Account` dropdown, choose the `New service account` option:
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_create_service_account_key.png" class="docs-image--no-shadow" caption="Create service account key" >}}
1. Some new fields will appear. Fill in a name for the service account in the `Service account name` field and then choose the `Monitoring Viewer` role from the `Role` dropdown:
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_service_account_choose_role.png" class="docs-image--no-shadow" caption="Choose role" >}}
1. Click the Create button. A JSON key file will be created and downloaded to your computer. Store this file in a secure place as it allows access to your Google Cloud Monitoring data.
1. Upload it to Grafana on the data source Configuration page. You can either upload the file or paste in the contents of the file.
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_grafana_upload_key.png" class="docs-image--no-shadow" caption="Upload service key file to Grafana" >}}
1. The file contents will be encrypted and saved in the Grafana database. Don't forget to save after uploading the file!
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_grafana_key_uploaded.png" class="docs-image--no-shadow" caption="Service key file is uploaded to Grafana" >}}
### Using GCE Default Service Account
If Grafana is running on a Google Compute Engine (GCE) virtual machine, it is possible for Grafana to automatically retrieve default credentials from the metadata server. This has the advantage of not needing to generate a private key file for the service account and also not having to upload the file to Grafana. However for this to work, there are a few preconditions that need to be met.
1. First of all, you need to create a Service Account that can be used by the GCE virtual machine. For more information, refer to [Create new service account](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#createanewserviceaccount).
1. Make sure the GCE virtual machine instance is being run as the service account that you just created. For more information, refer to [using service account for instance](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#using).
1. Allow access to the `Cloud Monitoring API` scope.
For more information about creating and enabling service accounts for GCE VM instances, refer to [enable service accounts for instances](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances).
## Using the Query Editor
The Google Cloud Monitoring query editor allows you to build two types of queries - **Metric** and **Service Level Objective (SLO)**. Both types return time series data.
### Metric Queries
{{< docs-imagebox img="/img/docs/v70/metric-query-builder.png" max-width= "400px" class="docs-image--right" >}}
The metric query editor allows you to select metrics, group/aggregate by labels and by time, and use filters to specify which time series you want in the results.
To create a metric query, follow these steps:
1. Choose the option **Metrics** in the **Query Type** dropdown
1. Choose a project from the **Project** dropdown
1. Choose a Google Cloud Platform service from the **Service** dropdown
1. Choose a metric from the **Metric** dropdown.
1. Use the plus and minus icons in the filter and group by sections to add/remove filters or group by clauses. This step is optional.
Google Cloud Monitoring metrics can be of different kinds (GAUGE, DELTA, CUMULATIVE) and these kinds have support for different aggregation options (reducers and aligners). The Grafana query editor shows the list of available aggregation methods for a selected metric and sets a default reducer and aligner when you select the metric. Units for the Y-axis are also automatically selected by the query editor.
#### Filter
To add a filter, click the plus icon and choose a field to filter by and enter a filter value e.g. `instance_name = grafana-1`. You can remove the filter by clicking on the filter name and select `--remove filter--`.
##### Simple wildcards
When the operator is set to `=` or `!=` it is possible to add wildcards to the filter value field. E.g `us-*` will capture all values that starts with "us-" and `*central-a` will capture all values that ends with "central-a". `*-central-*` captures all values that has the substring of -central-. Simple wildcards are less expensive than regular expressions.
##### Regular expressions
When the operator is set to `=~` or `!=~` it is possible to add regular expressions to the filter value field. E.g `us-central[1-3]-[af]` would match all values that starts with "us-central", is followed by a number in the range of 1 to 3, a dash and then either an "a" or an "f". Leading and trailing slashes are not needed when creating regular expressions.
#### Aggregation
The aggregation field lets you combine time series based on common statistics. For more information about aggregation, refer to [aggregation options](https://cloud.google.com/monitoring/charts/metrics-selector#aggregation-options).
The `Aligner` field allows you to align multiple time series after the same group by time interval. For more information about aligner, refer to [alignment metric selector](https://cloud.google.com/monitoring/charts/metrics-selector#alignment).
##### Alignment Period/Group by Time
The `Alignment Period` groups a metric by time if an aggregation is chosen. The default is to use the GCP Google Cloud Monitoring default groupings (which allows you to compare graphs in Grafana with graphs in the Google Cloud Monitoring UI).
The option is called `cloud monitoring auto` and the defaults are:
- 1m for time ranges < 23 hours
- 5m for time ranges >= 23 hours and < 6 days
- 1h for time ranges >= 6 days
The other automatic option is `grafana auto`. This will automatically set the group by time depending on the time range chosen and the width of the graph panel. For more information about grafana auto, refer to the [interval variable](http://docs.grafana.org/variables/templates-and-variables/#the-interval-variable).
It is also possible to choose fixed time intervals to group by, like `1h` or `1d`.
#### Group By
Group by resource or metric labels to reduce the number of time series and to aggregate the results by a group by. E.g. Group by instance_name to see an aggregated metric for a Compute instance.
##### Metadata labels
Resource metadata labels contain information to uniquely identify a resource in Google Cloud. Metadata labels are only returned in the time series response if they're part of the **Group By** segment in the time series request. There's no API for retrieving metadata labels, so it's not possible to populate the group by dropdown with the metadata labels that are available for the selected service and metric. However, the **Group By** field dropdown comes with a pre-defined list of common system labels.
User labels cannot be pre-defined, but it's possible to enter them manually in the **Group By** field. If a metadata label, user label or system label is included in the **Group By** segment, then you can create filters based on it and expand its value on the **Alias** field.
#### Alias patterns
The Alias By field allows you to control the format of the legend keys. The default is to show the metric name and labels. This can be long and hard to read. Using the following patterns in the alias field, you can format the legend key the way you want it.
#### Metric Type Patterns
| Alias Pattern | Description | Example Result |
| -------------------- | ---------------------------- | ------------------------------------------------- |
| `{{metric.type}}` | returns the full Metric Type | `compute.googleapis.com/instance/cpu/utilization` |
| `{{metric.name}}` | returns the metric name part | `instance/cpu/utilization` |
| `{{metric.service}}` | returns the service part | `compute` |
#### Label Patterns
In the Group By dropdown, you can see a list of metric and resource labels for a metric. These can be included in the legend key using alias patterns.
| Alias Pattern Format | Description | Alias Pattern Example | Example Result |
| -------------------------------- | ---------------------------------------- | --------------------------------- | ---------------- |
| `{{metric.label.xxx}}` | returns the metric label value | `{{metric.label.instance_name}}` | `grafana-1-prod` |
| `{{resource.label.xxx}}` | returns the resource label value | `{{resource.label.zone}}` | `us-east1-b` |
| `{{metadata.system_labels.xxx}}` | returns the meta data system label value | `{{metadata.system_labels.name}}` | `grafana` |
| `{{metadata.user_labels.xxx}}` | returns the meta data user label value | `{{metadata.user_labels.tag}}` | `production` |
Example Alias By: `{{metric.type}} - {{metric.labels.instance_name}}`
Example Result: `compute.googleapis.com/instance/cpu/usage_time - server1-prod`
It is also possible to resolve the name of the Monitored Resource Type.
| Alias Pattern Format | Description | Example Result |
| -------------------- | ----------------------------------------------- | -------------- |
| `{{resource.type}}` | returns the name of the monitored resource type | `gce_instance` |
Example Alias By: `{{resource.type}} - {{metric.type}}`
Example Result: `gce_instance - compute.googleapis.com/instance/cpu/usage_time`
### SLO (Service Level Objective) queries
> Only available in Grafana v7.0+
{{< docs-imagebox img="/img/docs/v70/slo-query-builder.png" max-width= "400px" class="docs-image--right" >}}
The SLO query builder in the Google Cloud Monitoring data source allows you to display SLO data in time series format. To get an understanding of the basic concepts in service monitoring, please refer to Google Cloud Monitoring's [official docs](https://cloud.google.com/monitoring/service-monitoring).
#### How to create an SLO query
To create an SLO query, follow these steps:
1. Choose the option **Service Level Objectives (SLO)** in the **Query Type** dropdown.
1. Choose a project from the **Project** dropdown.
1. Choose an [SLO service](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services) from the **Service** dropdown.
1. Choose an [SLO](https://cloud.google.com/monitoring/api/ref_v3/rest/v3/services.serviceLevelObjectives) from the **SLO** dropdown.
1. Choose a [time series selector](https://cloud.google.com/monitoring/service-monitoring/timeseries-selectors#ts-selector-list) from the **Selector** dropdown.
The friendly names for the time series selectors are shown in Grafana. Here is the mapping from the friendly name to the system name that is used in the Service Monitoring documentation:
| Selector dropdown value | Corresponding time series selector used |
| -------------------------- | --------------------------------------- |
| SLI Value | select_slo_health |
| SLO Compliance | select_slo_compliance |
| SLO Error Budget Remaining | select_slo_budget_fraction |
#### Alias Patterns for SLO queries
The Alias By field allows you to control the format of the legend keys for SLO queries too.
| Alias Pattern | Description | Example Result |
| -------------- | ---------------------------- | ------------------- |
| `{{project}}` | returns the GCP project name | `myProject` |
| `{{service}}` | returns the service name | `myService` |
| `{{slo}}` | returns the SLO | `latency-slo` |
| `{{selector}}` | returns the selector | `select_slo_health` |
#### Alignment Period/Group by Time for SLO queries
SLO queries use the same [alignment period functionality as metric queries]({{< relref "#metric-queries" >}}).
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place.
Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Query Variable
Variable of the type _Query_ allows you to query Google Cloud Monitoring for various types of data. The Google Cloud Monitoring data source plugin provides the following `Query Types`.
| Name | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------- |
| _Metric Types_ | Returns a list of metric type names that are available for the specified service. |
| _Labels Keys_ | Returns a list of keys for `metric label` and `resource label` in the specified metric. |
| _Labels Values_ | Returns a list of values for the label in the specified metric. |
| _Resource Types_ | Returns a list of resource types for the specified metric. |
| _Aggregations_ | Returns a list of aggregations (cross series reducers) for the specified metric. |
| _Aligners_ | Returns a list of aligners (per series aligners) for the specified metric. |
| _Alignment periods_ | Returns a list of all alignment periods that are available in Google Cloud Monitoring query editor in Grafana |
| _Selectors_ | Returns a list of selectors that can be used in SLO (Service Level Objectives) queries |
| _SLO Services_ | Returns a list of Service Monitoring services that can be used in SLO queries |
| _Service Level Objectives (SLO)_ | Returns a list of SLO's for the specified SLO service |
### Using variables in queries
There are two syntaxes:
- `$<varname>` Example: `metric.label.$metric_label`
- `[[varname]]` Example: `metric.label.[[metric_label]]`
Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the _Multi-value_ or _Include all value_ options are enabled, Grafana converts the labels from plain text to a regex compatible string, which means you have to use `=~` instead of `=`.
## Annotations
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_annotations_query_editor.png" max-width= "400px" class="docs-image--right" >}}
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation
queries via the Dashboard menu / Annotations view. Annotation rendering is expensive so it is important to limit the number of rows returned. There is no support for showing Google Cloud Monitoring annotations and events yet but it works well with [custom metrics](https://cloud.google.com/monitoring/custom-metrics/) in Google Cloud Monitoring.
With the query editor for annotations, you can select a metric and filters. The `Title` and `Text` fields support templating and can use data returned from the query. For example, the Title field could have the following text:
`{{metric.type}} has value: {{metric.value}}`
Example Result: `monitoring.googleapis.com/uptime_check/http_status has this value: 502`
### Patterns for the Annotation Query Editor
| Alias Pattern Format | Description | Alias Pattern Example | Example Result |
| ------------------------ | -------------------------------- | -------------------------------- | ------------------------------------------------- |
| `{{metric.value}}` | value of the metric/point | `{{metric.value}}` | `555` |
| `{{metric.type}}` | returns the full Metric Type | `{{metric.type}}` | `compute.googleapis.com/instance/cpu/utilization` |
| `{{metric.name}}` | returns the metric name part | `{{metric.name}}` | `instance/cpu/utilization` |
| `{{metric.service}}` | returns the service part | `{{metric.service}}` | `compute` |
| `{{metric.label.xxx}}` | returns the metric label value | `{{metric.label.instance_name}}` | `grafana-1-prod` |
| `{{resource.label.xxx}}` | returns the resource label value | `{{resource.label.zone}}` | `us-east1-b` |
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here is a provisioning example using the JWT (Service Account key file) authentication type.
```yaml
apiVersion: 1
datasources:
- name: Google Cloud Monitoring
type: stackdriver
access: proxy
jsonData:
tokenUri: https://oauth2.googleapis.com/token
clientEmail: stackdriver@myproject.iam.gserviceaccount.com
authenticationType: jwt
defaultProject: my-project-name
secureJsonData:
privateKey: |
-----BEGIN PRIVATE KEY-----
POSEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCb1u1Srw8ICYHS
...
yA+23427282348234=
-----END PRIVATE KEY-----
```
Here is a provisioning example using GCE Default Service Account authentication.
```yaml
apiVersion: 1
datasources:
- name: Google Cloud Monitoring
type: stackdriver
access: proxy
jsonData:
authenticationType: gce
```
## Deep linking from Grafana panels to the Metrics Explorer in Google Cloud Console
Only available in Grafana v7.1+.
{{< docs-imagebox img="/img/docs/v71/cloudmonitoring_deep_linking.png" max-width="500px" class="docs-image--right" caption="Google Cloud Monitoring deep linking" >}}
> **Note:** This feature is only available for Metric queries.
Click on a time series in the panel to see a context menu with a link to View in Metrics Explorer in Google Cloud Console. Clicking that link opens the Metrics Explorer in the Google Cloud Console and runs the query from the Grafana panel there.
The link navigates the user first to the Google Account Chooser and after successfully selecting an account, the user is redirected to the Metrics Explorer. The provided link is valid for any account, but it only displays the query if your account has access to the GCP project specified in the query.
docs/sources/datasources/cloudwatch.md
+404
View File
@@ -0,0 +1,404 @@
+++
title = "AWS CloudWatch"
description = "Guide for using CloudWatch in Grafana"
keywords = ["grafana", "cloudwatch", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/datasources/cloudwatch"]
[menu.docs]
name = "AWS Cloudwatch"
identifier = "cloudwatch"
parent = "datasources"
weight = 200
+++
# Using AWS CloudWatch in Grafana
Grafana ships with built-in support for CloudWatch. Add it as a data source, then you are ready to
build dashboards or use Explore with CloudWatch metrics and CloudWatch Logs.
## Adding the data source
1. In the side menu under the `Configuration` link, click on `Data Sources`.
1. Click the `Add data source` button.
1. Select `Cloudwatch` in the `Cloud` section.
> **Note:** If at any moment you have issues with getting this data source to work and Grafana is giving you undescriptive errors then don't
> forget to check your log file (try looking in /var/log/grafana/grafana.log).
| Name | Description |
| -------------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels and queries. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _Default Region_ | Used in query editor to set region (can be changed on per query basis) |
| _Custom Metrics namespace_ | Specify the CloudWatch namespace of Custom metrics |
| _Auth Provider_ | Specify the provider to get credentials. |
| _Credentials_ profile name | Specify the name of the profile to use (if you use `~/.aws/credentials` file), leave blank for default. |
| _Assume Role Arn_ | Specify the ARN of the role to assume |
| _External ID_ | If you are assuming a role in another account, that has been created with an external ID, specify the external ID here. |
## Authentication
### IAM Roles
Currently all access to CloudWatch is done server side by the Grafana backend using the official AWS SDK. If your Grafana
server is running on AWS you can use IAM Roles and authentication will be handled automatically.
See the AWS documentation on [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html)
> **Note:** [AWS Role Switching](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-cli.html) is not supported at the moment.
## IAM Policies
Grafana needs permissions granted via IAM to be able to read CloudWatch metrics
and EC2 tags/instances/regions. You can attach these permissions to IAM roles and
utilize Grafana's built-in support for assuming roles.
Here is a minimal policy example:
```bash
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowReadingMetricsFromCloudWatch",
"Effect": "Allow",
"Action": [
"cloudwatch:DescribeAlarmsForMetric",
"cloudwatch:DescribeAlarmHistory",
"cloudwatch:DescribeAlarms",
"cloudwatch:ListMetrics",
"cloudwatch:GetMetricStatistics",
"cloudwatch:GetMetricData"
],
"Resource": "*"
},
{
"Sid": "AllowReadingLogsFromCloudWatch",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:GetLogGroupFields",
"logs:StartQuery",
"logs:StopQuery",
"logs:GetQueryResults",
"logs:GetLogEvents"
],
"Resource": "*"
},
{
"Sid": "AllowReadingTagsInstancesRegionsFromEC2",
"Effect": "Allow",
"Action": ["ec2:DescribeTags", "ec2:DescribeInstances", "ec2:DescribeRegions"],
"Resource": "*"
},
{
"Sid": "AllowReadingResourcesForTags",
"Effect": "Allow",
"Action": "tag:GetResources",
"Resource": "*"
}
]
}
```
### AWS credentials
If Auth Provider is `Credentials file`, Grafana tries to get credentials in the following order.
- Environment variables. (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`)
- Hard-code credentials.
- Shared credentials file.
- IAM role for Amazon EC2.
See the AWS documentation on [Configuring the AWS SDK for Go](https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html)
### AWS credentials file
Create a file at `~/.aws/credentials`. That is the `HOME` path for user running grafana-server.
> **Note:** If you think you have the credentials file in the right place but it is still not working then you might try moving your .aws file to '/usr/share/grafana/' and make sure your credentials file has at most 0644 permissions.
Example content:
```bash
[default]
aws_access_key_id = asdsadasdasdasd
aws_secret_access_key = dasdasdsadasdasdasdsa
region = us-west-2
```
## Using the Query Editor
The CloudWatch data source can query data from both CloudWatch metrics and CloudWatch Logs APIs, each with its own specialized query editor. You select which API you want to query with using the query mode switch on top of the editor.
{{< docs-imagebox img="/img/docs/v70/cloudwatch-metrics-query-field.png" max-width="800px" class="docs-image--left" caption="CloudWatch metrics query field" >}}
{{< docs-imagebox img="/img/docs/v70/cloudwatch-logs-query-field.png" max-width="800px" class="docs-image--right" caption="CloudWatch Logs query field" >}}
## Using the Metric Query Editor
To create a valid query, you need to specify the namespace, metric name and at least one statistic. If `Match Exact` is enabled, you also need to specify all the dimensions of the metric you’re querying, so that the [metric schema](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html) matches exactly. If `Match Exact` is off, you can specify any number of dimensions by which you’d like to filter. Up to 100 metrics matching your filter criteria will be returned.
### Dynamic queries using dimension wildcards
> Only available in Grafana v6.5+.
In Grafana 6.5 or higher, you’re able to monitor a dynamic list of metrics by using the asterisk (\*) wildcard for one or more dimension values.
{{< docs-imagebox img="/img/docs/v65/cloudwatch-dimension-wildcard.png" max-width="800px" class="docs-image--right" caption="CloudWatch dimension wildcard" >}}
In the example, all metrics in the namespace `AWS/EC2` with a metric name of `CPUUtilization` and ANY value for the `InstanceId` dimension are queried. This can help you monitor metrics for AWS resources, like EC2 instances or containers. For example, when new instances get created as part of an auto scaling event, they will automatically appear in the graph without you having to track the new instance IDs. This capability is currently limited to retrieving up to 100 metrics. You can click on `Show Query Preview` to see the search expression that is automatically built to support wildcards. To learn more about search expressions, visit the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/search-expression-syntax.html).
By default, the search expression is defined in such a way that the queried metrics must match the defined dimension names exactly. This means that in the example only metrics with exactly one dimension with name ‘InstanceId’ will be returned.
You can untoggle `Match Exact` to include metrics that have other dimensions defined. Disabling `Match Exact` also creates a search expression even if you don’t use wildcards. We simply search for any metric that matches at least the namespace, metric name, and all defined dimensions.
### Multi-value template variables
> Only available in Grafana v6.5+.
When defining dimension values based on multi-valued template variables, a search expression is used to query for the matching metrics. This enables the use of multiple template variables in one query and also allows you to use template variables for queries that have the `Match Exact` option disabled.
Search expressions are currently limited to 1024 characters, so your query may fail if you have a long list of values. We recommend using the asterisk (`*`) wildcard instead of the `All` option if you want to query all metrics that have any value for a certain dimension name.
The use of multi-valued template variables is only supported for dimension values. Using multi-valued template variables for `Region`, `Namespace`, or `Metric Name` is not supported.
### Metric math expressions
You can create new time series metrics by operating on top of CloudWatch metrics using mathematical functions. Arithmetic operators, unary subtraction and other functions are supported and can be applied to CloudWatch metrics. More details on the available functions can be found on [AWS Metric Math](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html)
As an example, if you want to apply arithmetic operations on a metric, you can do it by giving an id (a unique string) to the raw metric as shown below. You can then use this id and apply arithmetic operations to it in the Expression field of the new metric.
Please note that in the case you use the expression field to reference another query, like `queryA * 2`, it will not be possible to create an alert rule based on that query.
### Period
A period is the length of time associated with a specific Amazon CloudWatch statistic. Periods are defined in numbers of seconds, and valid values for period are 1, 5, 10, 30, or any multiple of 60.
If the period field is left blank or set to `auto`, then it calculates automatically based on the time range. The formula used is `time range in seconds / 2000`, and then it snaps to the next higher value in an array of predefined periods `[60, 300, 900, 3600, 21600, 86400]`. By clicking `Show Query Preview` in the query editor, you can see what period Grafana used.
### Deep linking from Grafana panels to the CloudWatch console
> Only available in Grafana v6.5+.
{{< docs-imagebox img="/img/docs/v65/cloudwatch-deep-linking.png" max-width="500px" class="docs-image--right" caption="CloudWatch deep linking" >}}
Left clicking a time series in the panel shows a context menu with a link to `View in CloudWatch console`. Clicking that link will open a new tab that will take you to the CloudWatch console and display all the metrics for that query. If you're not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you're logged in to the account that corresponds to the selected data source in Grafana.
This feature is not available for metrics that are based on metric math expressions.
## Using the Logs Query Editor
> Only available in Grafana v7.0+.
To query CloudWatch Logs, select the region and up to 20 log groups which you want to query. Use the main input area to write your query in [CloudWatch Logs Query Language](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_QuerySyntax.html)
You can also write queries returning time series data by using the [`stats` command](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Visualizing-Log-Data.html). When making `stats` queries in Explore, you have to make sure you are in Metrics Explore mode.
{{< docs-imagebox img="/img/docs/v70/explore-mode-switcher.png" max-width="500px" class="docs-image--right" caption="Explore mode switcher" >}}
To the right of the query input field is a CloudWatch Logs Insights link that opens the CloudWatch Logs Insights console with your query. You can continue exploration there if necessary.
{{< docs-imagebox img="/img/docs/v70/cloudwatch-logs-deep-linking.png" max-width="500px" class="docs-image--right" caption="CloudWatch Logs deep linking" >}}
### Using template variables
As with several other data sources, the CloudWatch data source supports the use of template variables in queries.
See the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
### Deep linking from Grafana panels to the CloudWatch console
{{< docs-imagebox img="/img/docs/v70/cloudwatch-logs-deep-linking.png" max-width="500px" class="docs-image--right" caption="CloudWatch Logs deep linking" >}}
If you'd like to view your query in the CloudWatch Logs Insights console, simply click the `CloudWatch Logs Insights` button next to the query editor.
If you're not currently logged in to the CloudWatch console, the link will forward you to the login page. The provided link is valid for any account but will only display the right metrics if you're logged in to the account that corresponds to the selected data source in Grafana.
### Alerting
Since CloudWatch Logs queries can return numeric data, for example through the use of the `stats` command, alerts are supported.
See the [Alerting]({{< relref "../alerting/alerts-overview.md" >}}) documentation for more on Grafana alerts.
## Curated dashboards
> Only available in Grafana v6.5+.
The updated CloudWatch data source ships with pre-configured dashboards for five of the most popular AWS services:
- Amazon Elastic Compute Cloud `Amazon EC2`,
- Amazon Elastic Block Store `Amazon EBS`,
- AWS Lambda `AWS Lambda`,
- Amazon CloudWatch Logs `Amazon CloudWatch Logs`, and
- Amazon Relational Database Service `Amazon RDS`.
To import the pre-configured dashboards, go to the configuration page of your CloudWatch data source and click on the `Dashboards` tab. Click `Import` for the dashboard you would like to use. To customize the dashboard, we recommend saving the dashboard under a different name, because otherwise the dashboard will be overwritten when a new version of the dashboard is released.
{{< docs-imagebox img="/img/docs/v65/cloudwatch-dashboard-import.png" caption="CloudWatch dashboard import" >}}
## Templated queries
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.
See the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
### Query variable
The CloudWatch data source provides the following queries that you can specify in the `Query` field in the Variable edit view. They allow you to fill a variable's options list with things like `region`, `namespaces`, `metric names` and `dimension keys/values`.
In place of `region` you can specify `default` to use the default region configured in the data source for the query,
e.g. `metrics(AWS/DynamoDB, default)` or `dimension_values(default, ..., ..., ...)`.
Read more about the available dimensions in the [CloudWatch Metrics and Dimensions Reference](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CW_Support_For_AWS.html).
| Name | Description |
| ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _regions()_ | Returns a list of all AWS regions |
| _namespaces()_ | Returns a list of namespaces CloudWatch support. |
| _metrics(namespace, [region])_ | Returns a list of metrics in the namespace. (specify region or use "default" for custom metrics) |
| _dimension_\__keys(namespace)_ | Returns a list of dimension keys in the namespace. |
| _dimension_\__values(region, namespace, metric, dimension_\__key, [filters])_ | Returns a list of dimension values matching the specified `region`, `namespace`, `metric`, `dimension_key` or you can use dimension `filters` to get more specific result as well. |
| _ebs_\__volume_\__ids(region, instance_\__id)_ | Returns a list of volume ids matching the specified `region`, `instance_id`. |
| _ec2_\__instance_\__attribute(region, attribute_\__name, filters)_ | Returns a list of attributes matching the specified `region`, `attribute_name`, `filters`. |
| _resource_\__arns(region, resource_\__type, tags)_ | Returns a list of ARNs matching the specified `region`, `resource_type` and `tags`. |
| _statistics()_ | Returns a list of all the standard statistics |
For details about the metrics CloudWatch provides, please refer to the [CloudWatch documentation](https://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/CW_Support_For_AWS.html).
#### Examples templated queries
Example dimension queries which will return list of resources for individual AWS Services:
| Query | Service |
| -------------------------------------------------------------------------------------------------------------------------------- | ---------------- |
| _dimension_\__values(us-east-1,AWS/ELB,RequestCount,LoadBalancerName)_ | ELB |
| _dimension_\__values(us-east-1,AWS/ElastiCache,CPUUtilization,CacheClusterId)_ | ElastiCache |
| _dimension_\__values(us-east-1,AWS/Redshift,CPUUtilization,ClusterIdentifier)_ | RedShift |
| _dimension_\__values(us-east-1,AWS/RDS,CPUUtilization,DBInstanceIdentifier)_ | RDS |
| _dimension_\__values(us-east-1,AWS/S3,BucketSizeBytes,BucketName)_ | S3 |
| _dimension_\__values(us-east-1,CWAgent,disk_\__used_\__percent,device,{"InstanceId":"\$instance_\__id"})_ | CloudWatch Agent |
| _resource_\__arns(eu-west-1,elasticloadbalancing:loadbalancer,{"elasticbeanstalk:environment-name":["myApp-dev","myApp-prod"]})_ | ELB |
| _resource_\__arns(eu-west-1,ec2:instance,{"elasticbeanstalk:environment-name":["myApp-dev","myApp-prod"]})_ | EC2 |
## ec2_instance_attribute examples
### JSON filters
The `ec2_instance_attribute` query takes `filters` in JSON format.
You can specify [pre-defined filters of ec2:DescribeInstances](http://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html).
Note that the actual filtering takes place on Amazon's servers, not in Grafana.
Filters syntax:
```javascript
{ filter_name1: [ filter_value1 ], filter_name2: [ filter_value2 ] }
```
Example `ec2_instance_attribute()` query
```javascript
ec2_instance_attribute(us - east - 1, InstanceId, { 'tag:Environment': ['production'] });
```
### Selecting attributes
Only 1 attribute per instance can be returned. Any flat attribute can be selected (i.e. if the attribute has a single value and isn't an object or array). Below is a list of available flat attributes:
- `AmiLaunchIndex`
- `Architecture`
- `ClientToken`
- `EbsOptimized`
- `EnaSupport`
- `Hypervisor`
- `IamInstanceProfile`
- `ImageId`
- `InstanceId`
- `InstanceLifecycle`
- `InstanceType`
- `KernelId`
- `KeyName`
- `LaunchTime`
- `Platform`
- `PrivateDnsName`
- `PrivateIpAddress`
- `PublicDnsName`
- `PublicIpAddress`
- `RamdiskId`
- `RootDeviceName`
- `RootDeviceType`
- `SourceDestCheck`
- `SpotInstanceRequestId`
- `SriovNetSupport`
- `SubnetId`
- `VirtualizationType`
- `VpcId`
Tags can be selected by prepending the tag name with `Tags.`
Example `ec2_instance_attribute()` query
```javascript
ec2_instance_attribute(us - east - 1, Tags.Name, { 'tag:Team': ['sysops'] });
```
## Using json format template variables
Some queries accept filters in JSON format and Grafana supports the conversion of template variables to JSON.
If `env = 'production', 'staging'`, following query will return ARNs of EC2 instances which `Environment` tag is `production` or `staging`.
```
resource_arns(us-east-1, ec2:instance, {"Environment":${env:json}})
```
## Pricing
The Amazon CloudWatch data source for Grafana uses the `ListMetrics` and `GetMetricData` CloudWatch API calls to list and retrieve metrics.
Pricing for CloudWatch Logs is based on the amount of data ingested, archived, and analyzed via CloudWatch Logs Insights queries.
Please see the [CloudWatch pricing page](https://aws.amazon.com/cloudwatch/pricing/) for more details.
Every time you pick a dimension in the query editor Grafana will issue a ListMetrics request.
Whenever you make a change to the queries in the query editor, one new request to GetMetricData will be issued.
Please note that for Grafana version 6.5 or higher, all API requests to GetMetricStatistics have been replaced with calls to GetMetricData. This change enables better support for CloudWatch metric math and enables the automatic generation of search expressions when using wildcards or disabling the `Match Exact` option. While GetMetricStatistics qualified for the CloudWatch API free tier, this is not the case for GetMetricData calls. For more information, please refer to the [CloudWatch pricing page](https://aws.amazon.com/cloudwatch/pricing/).
## Service quotas
AWS defines quotas, or limits, for resources, actions, and items in your AWS account. Depending on the number of queries in your dashboard and the number of users accessing the dashboard, you may reach the usage limits for various CloudWatch and CloudWatch Logs resources. Note that quotas are defined per account and per region. If you're using multiple regions or have set up more than one CloudWatch data source to query against multiple accounts, you need to request a quota increase for each account and each region in which you hit the limit.
To request a quota increase, visit the [AWS Service Quotas console](https://console.aws.amazon.com/servicequotas/home?r#!/services/monitoring/quotas/L-5E141212).
Please see the AWS documentation for [Service Quotas](https://docs.aws.amazon.com/servicequotas/latest/userguide/intro.html) and [CloudWatch limits](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_limits.html) for more information.
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
### Using credentials profile name (non-default)
```yaml
apiVersion: 1
datasources:
- name: Cloudwatch
type: cloudwatch
jsonData:
authType: credentials
defaultRegion: eu-west-2
customMetricsNamespaces: 'CWAgent,CustomNameSpace'
profile: secondary
```
### Using `accessKey` and `secretKey`
```yaml
apiVersion: 1
datasources:
- name: Cloudwatch
type: cloudwatch
jsonData:
authType: keys
defaultRegion: eu-west-2
secureJsonData:
accessKey: '<your access key>'
secretKey: '<your secret key>'
```
docs/sources/datasources/elasticsearch.md
+250
View File
@@ -0,0 +1,250 @@
+++
title = "Elasticsearch"
description = "Guide for using Elasticsearch in Grafana"
keywords = ["grafana", "elasticsearch", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/elasticsearch"]
[menu.docs]
name = "Elasticsearch"
parent = "datasources"
weight = 400
+++
# Using Elasticsearch in Grafana
Grafana ships with advanced support for Elasticsearch. You can do many types of simple or complex Elasticsearch queries to
visualize logs or metrics stored in Elasticsearch. You can also annotate your graphs with log events stored in Elasticsearch.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Dashboards` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *Elasticsearch* from the *Type* dropdown.
> **Note:** If you're not seeing the `Data Sources` link in your side menu it means that your current user does not have the `Admin` role for the current organization.
| Name | Description |
| --------- | ------------------------------------------------------------------------------------------------------------------------------------- |
| *Name* | The data source name. This is how you refer to the data source in panels and queries. |
| *Default* | Default data source means that it will be pre-selected for new panels. |
| *Url* | The HTTP protocol, IP, and port of your Elasticsearch server. |
| *Access* | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. |
Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else stated.
### Server access mode (Default)
All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the grafana backend/server if you select this access mode.
### Browser (Direct) access
All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.
If you select Browser access you must update your Elasticsearch configuration to allow other domains to access
Elasticsearch from the browser. You do this by specifying these two options in your **elasticsearch.yml** config file.
```bash
http.cors.enabled: true
http.cors.allow-origin: "*"
```
### Index settings
![Elasticsearch data source details](/img/docs/elasticsearch/elasticsearch_ds_details.png)
Here you can specify a default for the `time field` and specify the name of your Elasticsearch index. You can use
a time pattern for the index name or a wildcard.
### Elasticsearch version
Be sure to specify your Elasticsearch version in the version selection dropdown. This is very important as there are differences on how queries are composed.
Currently the versions available are `2.x`, `5.x`, `5.6+`, `6.0+` or `7.0+`. The value `5.6+` means version 5.6 or higher, but lower than 6.0. The value `6.0+` means
version 6.0 or higher, but lower than 7.0. Finally, `7.0+` means version 7.0 or higher, but lower than 8.0.
### Min time interval
A lower limit for the auto group by time interval. Recommended to be set to write frequency, for example `1m` if your data is written every minute.
This option can also be overridden/configured in a dashboard panel under data source options. It's important to note that this value **needs** to be formatted as a
number followed by a valid time identifier, e.g. `1m` (1 minute) or `30s` (30 seconds). The following time identifiers are supported:
| Identifier | Description |
| ---------- | ----------- |
| `y` | year |
| `M` | month |
| `w` | week |
| `d` | day |
| `h` | hour |
| `m` | minute |
| `s` | second |
| `ms` | millisecond |
### Logs (BETA)
> Only available in Grafana v6.3+.
There are two parameters, `Message field name` and `Level field name`, that can optionally be configured from the data source settings page that determine
which fields will be used for log messages and log levels when visualizing logs in [Explore]({{< relref "../explore" >}}).
For example, if you're using a default setup of Filebeat for shipping logs to Elasticsearch the following configuration should work:
- **Message field name:** message
- **Level field name:** fields.level
### Data links
Data links create a link from a specified field that can be accessed in logs view in Explore.
Each data link configuration consists of:
- **Field -** Name of the field used by the data link.
- **URL/query -** If the link is external, then enter the full link URL. If the link is internal link, then this input serves as query for the target data source. In both cases, you can interpolate the value from the field with `${__value.raw }` macro.
- **Internal link -** Select if the link is internal or external. In case of internal link, a data source selector allows you to select the target data source. Only tracing data sources are supported.
## Metric Query editor
![Elasticsearch Query Editor](/img/docs/elasticsearch/query_editor.png)
The Elasticsearch query editor allows you to select multiple metrics and group by multiple terms or filters. Use the plus and minus icons to the right to add/remove
metrics or group by clauses. Some metrics and group by clauses haves options, click the option text to expand the row to view and edit metric or group by options.
## Series naming and alias patterns
You can control the name for time series via the `Alias` input field.
| Pattern | Description |
| -------------------- | ------------------------------------------------- |
| *{{term fieldname}}* | replaced with value of a term group by |
| *{{metric}}* | replaced with metric name (ex. Average, Min, Max) |
| *{{field}}* | replaced with the metric field name |
## Pipeline metrics
Some metric aggregations are called Pipeline aggregations, for example, *Moving Average* and *Derivative*. Elasticsearch pipeline metrics require another metric to be based on. Use the eye icon next to the metric to hide metrics from appearing in the graph. This is useful for metrics you only have in the query for use in a pipeline metric.
![](/img/docs/elasticsearch/pipeline_metrics_editor.png)
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place.
Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Query variable
The Elasticsearch data source supports two types of queries you can use in the *Query* field of *Query* variables. The query is written using a custom JSON string.
| Query | Description |
| -------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| *{"find": "fields", "type": "keyword"}* | Returns a list of field names with the index type `keyword`. |
| *{"find": "terms", "field": "@hostname", "size": 1000}* | Returns a list of values for a field using term aggregation. Query will use current dashboard time range as time range for query. |
| *{"find": "terms", "field": "@hostname", "query": '<lucene query>'}* | Returns a list of values for a field using term aggregation and a specified lucene query filter. Query will use current dashboard time range as time range for query. |
There is a default size limit of 500 on terms queries. Set the size property in your query to set a custom limit.
You can use other variables inside the query. Example query definition for a variable named `$host`.
```
{"find": "terms", "field": "@hostname", "query": "@source:$source"}
```
In the above example, we use another variable named `$source` inside the query definition. Whenever you change, via the dropdown, the current value of the ` $source` variable, it will trigger an update of the `$host` variable so it now only contains hostnames filtered by in this case the
`@source` document property.
These queries by default return results in term order (which can then be sorted alphabetically or numerically as for any variable).
To produce a list of terms sorted by doc count (a top-N values list), add an `orderBy` property of "doc_count".
This automatically selects a descending sort; using "asc" with doc_count (a bottom-N list) can be done by setting `order: "asc"` but [is discouraged](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-order) as it "increases the error on document counts".
To keep terms in the doc count order, set the variable's Sort dropdown to **Disabled**; you might alternatively still want to use e.g. **Alphabetical** to re-sort them.
```
{"find": "terms", "field": "@hostname", "orderBy": "doc_count"}
```
### Using variables in queries
There are two syntaxes:
- `$<varname>` Example: @hostname:$hostname
- `[[varname]]` Example: @hostname:[[hostname]]
Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the *Multi-value* or *Include all value*
options are enabled, Grafana converts the labels from plain text to a lucene compatible condition.
![](/img/docs/v43/elastic_templating_query.png)
In the above example, we have a lucene query that filters documents based on the `@hostname` property using a variable named `$hostname`. It is also using
a variable in the *Terms* group by field input box. This allows you to use a variable to quickly change how the data is grouped.
Example dashboard:
[Elasticsearch Templated Dashboard](https://play.grafana.org/dashboard/db/elasticsearch-templated)
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation
queries via the Dashboard menu / Annotations view. Grafana can query any Elasticsearch index
for annotation events.
| Name | Description |
| -------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
| Query | You can leave the search query blank or specify a lucene query. |
| Time | The name of the time field, needs to be date field. |
| Time End | Optional name of the time end field needs to be date field. If set, then annotations will be marked as a region between time and time-end. |
| Text | Event description field. |
| Tags | Optional field name to use for event tags (can be an array or a CSV string). |
## Querying Logs (BETA)
> Only available in Grafana v6.3+.
Querying and displaying log data from Elasticsearch is available in [Explore]({{< relref "../explore" >}}), and in the [logs panel]({{< relref "../panels/visualizations/logs-panel.md" >}}) in dashboards.
Select the Elasticsearch data source, and then optionally enter a lucene query to display your logs.
### Log Queries
Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
Note that the fields used for log message and level is based on an [optional data source configuration](#logs-beta).
### Filter Log Messages
Optionally enter a lucene query into the query field to filter the log messages. For example, using a default Filebeat setup you should be able to use `fields.level:error` to only show error log messages.
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: Elastic
type: elasticsearch
access: proxy
database: "[metrics-]YYYY.MM.DD"
url: http://localhost:9200
jsonData:
interval: Daily
timeField: "@timestamp"
```
or, for logs:
```yaml
apiVersion: 1
datasources:
- name: elasticsearch-v7-filebeat
type: elasticsearch
access: proxy
database: "[filebeat-]YYYY.MM.DD"
url: http://localhost:9200
jsonData:
interval: Daily
timeField: "@timestamp"
esVersion: 70
logMessageField: message
logLevelField: fields.level
```
docs/sources/datasources/graphite.md
+224
View File
@@ -0,0 +1,224 @@
+++
title = "Graphite"
description = "Guide for using graphite in Grafana"
keywords = ["grafana", "graphite", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/graphite"]
[menu.docs]
name = "Graphite"
identifier = "graphite"
parent = "datasources"
weight = 600
+++
# Using Graphite in Grafana
Grafana has an advanced Graphite query editor that lets you quickly navigate the metric space, add functions,
change function parameters and much more. The editor can handle all types of graphite queries. It can even handle complex nested
queries through the use of query references.
Refer to [Add a data source]({{< relref "add-a-data-source.md" >}}) for instructions on how to add a data source to Grafana. Only organization admins can add data sources.
## Graphite settings
To access Graphite settings, hover your mouse over the **Configuration** (gear) icon, then click **Data Sources**, and then click the Graphite data source.
Name | Description
------------ | -------------
Name | The data source name. This is how you refer to the data source in panels and queries.
Default | Default data source means that it will be pre-selected for new panels.
URL | The HTTP protocol, IP, and port of your graphite-web or graphite-api install.
Access | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser.
Auth | Refer to [Authentication]({{< relref "../auth/_index.md" >}}) for more information.
Basic Auth | Enable basic authentication to the data source.
User | User name for basic authentication.
Password | Password for basic authentication.
Custom HTTP Headers | Click **Add header** to add a custom HTTP header.
Header | Enter the custom header name.
Value | Enter the custom header value.
Graphite details |
Version | Select your version of Graphite.
Type | Select your type of Graphite.
Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else is stated.
### Server access mode (default)
All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the Grafana backend/server if you select this access mode.
### Browser access mode
All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.
## Graphite query editor
Grafana includes a Graphite-specific query editor to help you build your queries.
To see the raw text of the query that is sent to Graphite, click the **Toggle text edit mode** (pencil) icon.
### Choose metrics to query
Click **Select metric** to start navigating the metric space. Once you start, you can continue using the mouse or keyboard arrow keys. You can select a wildcard and still continue.
{{< docs-imagebox img="/img/docs/graphite/graphite-query-editor-still.png"
animated-gif="/img/docs/graphite/graphite-query-editor.gif" >}}
### Functions
Click the plus icon next to **Function** to add a function. You can search for the function or select it from the menu. Once
a function is selected, it will be added and your focus will be in the text box of the first parameter.
- To edit or change a parameter, click on it and it will turn into a text box.
- To delete a function, click the function name followed by the x icon.
{{< docs-imagebox img="/img/docs/graphite/graphite-functions-still.png"
animated-gif="/img/docs/graphite/graphite-functions-demo.gif" >}}
Some functions like aliasByNode support an optional second argument. To add an argument, hover your mouse over the first argument and then click the `+` symbol that appears. To remove the second optional parameter, click on it and leave it blank and the editor will remove it.
### Sort labels
If you want consistent ordering, use sortByName. This can be particularly annoying when you have the same labels on multiple graphs, and they are both sorted differently and using different colors. To fix this, use `sortByName()`.
### Nested queries
You can reference queries by the row “letter” that they’re on (similar to Microsoft Excel). If you add a second query to a graph, you can reference the first query simply by typing in #A. This provides an easy and convenient way to build compounded queries.
### Avoiding many queries by using wildcards
Occasionally one would like to see multiple time series plotted on the same graph. For example we might want to see how the CPU is being utilized on a machine. You might
initially create the graph by adding a query for each time series, such as `cpu.percent.user.g`,
`cpu.percent.system.g`, and so on. This results in *n* queries made to the data source, which is inefficient.
To be more efficient one can use wildcards in your search, returning all the time series in one query. For example, `cpu.percent.*.g`.
### Modify the metric name in my tables or charts
Use `alias` functions to change metric names on Grafana tables or graphs For example `aliasByNode()` or `aliasSub()`.
## Point consolidation
All Graphite metrics are consolidated so that Graphite doesn't return more data points than there are pixels in the graph. By default,
this consolidation is done using `avg` function. You can control how Graphite consolidates metrics by adding the Graphite consolidateBy function.
> **Note:** This means that legend summary values (max, min, total) cannot all be correct at the same time. They are calculated
> client-side by Grafana. And depending on your consolidation function, only one or two can be correct at the same time.
## Combine time series
To combine time series, click **Combine** in the **Functions** list.
## Data exploration and tags
In Graphite, _everything_ is a tag.
When exploring data, previously-selected tags are used to filter the remaining result set. To select data, you use the
`seriesByTag` function, which takes tag expressions (`=`, `!=`, `=~`, `!=~`) to filter timeseries.
The Grafana query builder does this for you automatically when you select a tag.
> **Tip:** The regular expression search can be quite slow on high-cardinality tags, so try to use other tags to reduce the scope first.
Starting off with a particular name/namespace can help reduce the results.
## Template variables
Instead of hard-coding things like server, application, and sensor name in your metric queries, you can use variables in their place.
Variables are shown as drop-down select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
For more information, refer to [Variables and templates]({{< relref "../variables/_index.md" >}}).
Graphite 1.1 introduced tags and Grafana added support for Graphite queries with tags in version 5.0. To create a variable using tag values, use the Grafana functions `tags` and `tag_values`.
Query | Description
------------ | -------------
tags() | Returns all tags.
tags(server=~backend\*) | Returns only tags that occur in series matching the filter expression.
tag_values(server) | Return tag values for the specified tag.
tag_values(server, server=~backend\*) | Returns filtered tag values that occur for the specified tag in series matching those expressions.
tag_values(server, server=~backend\*, app=~${apps:regex}) | Multiple filter expressions and expressions can contain other variables.
For more details, see the [Graphite docs on the autocomplete API for tags](http://graphite.readthedocs.io/en/latest/tags.html#auto-complete-support).
### Query variable
The query you specify in the query field should be a metric find type of query. For example, a query like `prod.servers.*` will fill the
variable with all possible values that exist in the wildcard position.
You can also create nested variables that use other variables in their definition. For example
`apps.$app.servers.*` uses the variable `$app` in its query definition.
#### Using `__searchFilter` to filter query variable results
> Available from Grafana 6.5 and above
Using `__searchFilter` in the query field will filter the query result based on what the user types in the dropdown select box.
When nothing has been entered by the user the default value for `__searchFilter` is `*` and `` when used as part of a regular expression.
The example below shows how to use `__searchFilter` as part of the query field to enable searching for `server` while the user types in the dropdown select box.
Query
```bash
apps.$app.servers.$__searchFilter
```
TagValues
```bash
tag_values(server, server=~${__searchFilter:regex})
```
### Variable usage
You can use a variable in a metric node path or as a parameter to a function.
![variable](/img/docs/v2/templated_variable_parameter.png)
There are two syntaxes:
- `$<varname>` Example: apps.frontend.$server.requests.count
- `${varname}` Example: apps.frontend.${server}.requests.count
Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. Use
the second syntax in expressions like `my.server${serverNumber}.count`.
Example:
[Graphite Templated Dashboard](https://play.grafana.org/dashboard/db/graphite-templated-nested)
### Variable usage in tag queries
Multi-value variables in tag queries use the advanced formatting syntax introduced in Grafana 5.0 for variables: `{var:regex}`. Non-tag queries will use the default glob formatting for multi-value variables.
Example of a tag expression with regex formatting and using the Equal Tilde operator, `=~`:
```text
server=~${servers:regex}
```
For more information, refer to [Advanced variable format options]({{< relref "../variables/advanced-variable-format-options.md" >}}).
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation
queries via the Dashboard menu / Annotations view.
Graphite supports two ways to query annotations. A regular metric query, for this you use the `Graphite query` textbox. A Graphite events query, use the `Graphite event tags` textbox,
specify a tag or wildcard (leave empty should also work)
## Get Grafana metrics into Graphite
Grafana exposes metrics for Graphite on the `/metrics` endpoint. For detailed instructions, refer to [Internal Grafana metrics]({{< relref "../administration/metrics.md">}}).
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: Graphite
type: graphite
access: proxy
url: http://localhost:8080
jsonData:
graphiteVersion: "1.1"
```
docs/sources/datasources/influxdb.md
+309
View File
@@ -0,0 +1,309 @@
+++
title = "InfluxDB"
description = "Guide for using InfluxDB in Grafana"
keywords = ["grafana", "influxdb", "guide", "flux"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/influxdb"]
[menu.docs]
name = "InfluxDB"
parent = "datasources"
weight = 700
+++
# Using InfluxDB in Grafana
Grafana ships with a feature-rich data source plugin for InfluxDB. The plugin includes a custom query editor and supports annotations and query templates.
## Add the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Dashboards` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *InfluxDB* from the *Type* dropdown.
1. Select *InfluxQL* or *Flux* from the `Query Language` list.
> **Note:** If you're not seeing the `Data Sources` link in your side menu it means that your current user does not have the `Admin` role for the current organization.
### InfluxQL (classic InfluxDB query)
Name | Description
------------ | -------------
*Name* | The data source name. This is how you refer to the data source in panels and queries.
*Default* | Default data source means that it will be pre-selected for new panels.
*URL* | The HTTP protocol, IP address and port of your InfluxDB API (InfluxDB API port is by default 8086)
*Access* | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser.
*Database* | Name of your InfluxDB database
*User* | Name of your database user
*Password* | Database user's password
*HTTP mode* | How to query the database (`GET` or `POST` HTTP verb). The `POST` verb allows heavy queries that would return an error using the `GET` verb. Default is `GET`.
Access mode controls how requests to the data source will be handled. Server should be the preferred way if nothing else stated.
### Server access mode (Default)
All requests will be made from the browser to Grafana backend/server which in turn will forward the requests to the data source and by that circumvent possible Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the grafana backend/server if you select this access mode.
### Browser access mode
All requests will be made from the browser directly to the data source and may be subject to Cross-Origin Resource Sharing (CORS) requirements. The URL needs to be accessible from the browser if you select this access mode.
### Min time interval
A lower limit for the auto group by time interval. Recommended to be set to write frequency, for example `1m` if your data is written every minute.
This option can also be overridden/configured in a dashboard panel under data source options. It's important to note that this value **needs** to be formatted as a
number followed by a valid time identifier, e.g. `1m` (1 minute) or `30s` (30 seconds). The following time identifiers are supported:
Identifier | Description
------------ | -------------
`y` | year
`M` | month
`w` | week
`d` | day
`h` | hour
`m` | minute
`s` | second
`ms` | millisecond
## Query Editor
{{< docs-imagebox img="/img/docs/v45/influxdb_query_still.png" class="docs-image--no-shadow" animated-gif="/img/docs/v45/influxdb_query.gif" >}}
You can access the InfluxDB editor under the metrics tab when you are in the edit mode of the Graph or Singlestat panels.
Enter edit mode by clicking the panel title and clicking **Edit**. The editor allows you to select metrics and tags.
### Filter data (WHERE)
To add a tag filter, click the plus icon to the right of the `WHERE` condition. You can remove tag filters by clicking on
the tag key and then selecting `--remove tag filter--`.
**Regex matching**
You can type in regex patterns for metric names or tag filter values. Be sure to wrap the regex pattern in forward slashes (`/`). Grafana automatically adjusts the filter tag condition to use the InfluxDB regex match condition operator (`=~`).
### Field and Aggregation functions
In the `SELECT` row you can specify what fields and functions you want to use. If you have a
group by time you need an aggregation function. Some functions like derivative require an aggregation function. The editor tries to simplify and unify this part of the query. For example:
![](/img/docs/influxdb/select_editor.png)
The above generates the following InfluxDB `SELECT` clause:
```sql
SELECT derivative(mean("value"), 10s) /10 AS "REQ/s" FROM ....
```
#### Select multiple fields
Use the plus button and select Field > field to add another SELECT clause. You can also
specify an asterix `*` to select all fields.
### Group By
To group by a tag, click the plus icon at the end of the GROUP BY row. Pick a tag from the dropdown that appears.
You can remove the "Group By" by clicking on the `tag` and then click on the x icon.
### Text Editor Mode (RAW)
You can switch to raw query mode by clicking hamburger icon and then `Switch editor mode`.
> If you use Raw Query be sure your query at minimum have `WHERE $timeFilter`
> Also please always have a group by time and an aggregation function, otherwise InfluxDB can easily return hundreds of thousands
> of data points that will hang the browser.
### Alias patterns
- $m = replaced with measurement name
- $measurement = replaced with measurement name
- $col = replaced with column name
- $tag_exampletag = replaced with the value of the `exampletag` tag. The syntax is `$tag_yourTagName` (must start with `$tag_`). To use your tag as an alias in the ALIAS BY field then the tag must be used to group by in the query.
- You can also use [[tag_hostname]] pattern replacement syntax. For example, in the ALIAS BY field using this text `Host: [[tag_hostname]]` would substitute in the `hostname` tag value for each legend value and an example legend value would be: `Host: server1`.
### Table query / raw data
![](/assets/img/blog/v2.6/table_influxdb_logs.png)
You can remove the group by time by clicking on the `time` part and then the `x` icon. You can
change the option `Format As` to `Table` if you want to show raw data in the `Table` panel.
## Flux support
> Starting in v7.1, Grafana can execute Flux queries.
The client supports Flux running on InfluxDB 1.8+. See [1.8 compatibility](https://github.com/influxdata/influxdb-client-go/#influxdb-18-api-compatibility) for more information and connection details.
Name | Description
------------ | -------------
*URL* | The HTTP protocol, IP address and port of your InfluxDB API (InfluxDB 2.0 API port is by default 9999)
*Organization* | The [Influx organization](https://v2.docs.influxdata.com/v2.0/organizations/) that will be used for Flux queries. This is also used to for the `v.organization` query macro
*Token* | The authentication token used for Flux queries. With Influx 2.0, use the [influx authentication token to function](https://v2.docs.influxdata.com/v2.0/security/tokens/create-token/). For influx 1.8, the token is `username:password`
*Default Bucket* | The [Influx bucket](https://v2.docs.influxdata.com/v2.0/organizations/buckets/) that will be used for the `v.defaultBucket` macro in Flux queries
You can use the [Flux query and scripting language](https://www.influxdata.com/products/flux/). Grafana's Flux query editor is a text editor for raw Flux queries with Macro support.
### Supported macros
The macros support copying and pasting from [Chronograph](https://www.influxdata.com/time-series-platform/chronograf/).
Macro example | Description
------------ | -------------
*`v.timeRangeStart`* | Will be replaced by the start of the currently active time selection. For example, *2020-06-11T13:31:00Z*
*`v.timeRangeEnd`* | Will be replaced by the end of the currently active time selection. For example, *2020-06-11T14:31:00Z*
*`v.windowPeriod`* | Will be replaced with an interval string compatible with Flux that corresponds to Grafana's calculated interval based on the time range of the active time selection. For example, *5s*
*`v.defaultBucket`* | Will be replaced with the data source configuration's "Default Bucket" setting
*`v.organization`* | Will be replaced with the data source configuration's "Organization" setting
For example, the following query will be interpolated as the query that follows it, with interval and time period values changing according to active time selection\):
Grafana Flux query:
```flux
from(bucket: v.defaultBucket)
|> range(start: v.timeRangeStart, stop: v.timeRangeStop)
|> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap")
|> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free")
|> aggregateWindow(every: v.windowPeriod, fn: mean)
|> yield(name: "mean")
```
Interpolated query send to Influx:
```flux
from(bucket: "grafana")
|> range(start: 2020-06-11T13:59:07Z, stop: 2020-06-11T14:59:07Z)
|> filter(fn: (r) => r["_measurement"] == "cpu" or r["_measurement"] == "swap")
|> filter(fn: (r) => r["_field"] == "usage_system" or r["_field"] == "free")
|> aggregateWindow(every: 2s, fn: mean)
|> yield(name: "mean")
```
You can view the interpolated version of a query with the Query Inspector.
## Querying Logs (BETA)
> Only available in Grafana v6.3+.
Querying and displaying log data from InfluxDB is available in [Explore]({{< relref "../explore" >}}), and in the [logs panel]({{< relref "../panels/visualizations/logs-panel.md" >}}) in dashboards.
Select the InfluxDB data source, and then enter a query to display your logs.
### Log Queries
The Logs Explorer (the `Measurements/Fields` button) next to the query field shows a list of measurements and fields. Choose the desired measurement that contains your log data and then choose which field Explore should use to display the log message.
Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
### Filter search
To add a filter, click the plus icon to the right of the `Measurements/Fields` button or a condition. You can remove tag filters by clicking on the first select and choosing `--remove filter--`.
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place.
Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Query variable
If you add a template variable of the type `Query`, then you can write an InfluxDB exploration (metadata) query. These queries can
return things like measurement names, key names or key values.
For example, you can have a variable that contains all values for tag `hostname` if you specify a query like this in the template variable *Query* setting.
```sql
SHOW TAG VALUES WITH KEY = "hostname"
```
You can also create nested variables. For example, if you had another variable, for example `region`. Then you could have
the hosts variable only show hosts from the current selected region with a query like this:
```sql
SHOW TAG VALUES WITH KEY = "hostname" WHERE region = '$region'
```
You can fetch key names for a given measurement.
```sql
SHOW TAG KEYS [FROM <measurement_name>]
```
If you have a variable with key names you can use this variable in a group by clause. This will allow you to change group by using the variable dropdown at the top
of the dashboard.
### Using variables in queries
There are two syntaxes:
`$<varname>` Example:
```sql
SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^$host$/ AND $timeFilter GROUP BY time($__interval), "hostname"
```
`[[varname]]` Example:
```sql
SELECT mean("value") FROM "logins" WHERE "hostname" =~ /^[[host]]$/ AND $timeFilter GROUP BY time($__interval), "hostname"
```
Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the *Multi-value* or *Include all value*
options are enabled, Grafana converts the labels from plain text to a regex compatible string. Which means you have to use `=~` instead of `=`.
Example Dashboard:
[InfluxDB Templated Dashboard](https://play.grafana.org/dashboard/db/influxdb-templated)
### Ad hoc filters variable
InfluxDB supports the special `Ad hoc filters` variable type. This variable allows you to specify any number of key/value filters on the fly. These filters will automatically
be applied to all your InfluxDB queries.
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allows you to overlay rich event information on top of graphs. Add annotation queries using the Annotations view in the Dashboard menu.
An example query:
```SQL
SELECT title, description from events WHERE $timeFilter ORDER BY time ASC
```
For InfluxDB, you need to enter a query like the one in the example above. The ```where $timeFilter``` component is required. If you only select one column, then you do not need to enter anything in the column mapping fields. The **Tags** field can be a comma-separated string.
## Configure the data source with provisioning
You can now configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}}).
Here are some provisioning examples for this data source.
### InfluxDB 1.x example
```yaml
apiVersion: 1
datasources:
- name: InfluxDB_v1
type: influxdb
access: proxy
database: site
user: grafana
password: grafana
url: http://localhost:8086
jsonData:
httpMode: GET
```
### InfluxDB 2.x example
```yaml
apiVersion: 1
datasources:
- name: InfluxDB_v2
type: influxdb
access: proxy
url: http://localhost:8086
secureJsonData:
token: token
jsonData:
version: Flux
organization: organization
defaultBucket: bucket
tlsSkipVerify: true
```
docs/sources/datasources/jaeger.md
+48
View File
@@ -0,0 +1,48 @@
+++
title = "Jaeger"
description = "Guide for using Jaeger in Grafana"
keywords = ["grafana", "jaeger", "guide", "tracing"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/jaeger"]
[menu.docs]
name = "Jaeger"
parent = "datasources"
weight = 800
+++
# Jaeger data source
Grafana ships with built-in support for Jaeger, which provides open source, end-to-end distributed tracing.
Just add it as a data source and you are ready to query your traces in [Explore]({{< relref "../explore/index.md" >}}).
## Adding the data source
To access Jaeger settings, click the **Configuration** (gear) icon, then click **Data Sources**, and then click **Jaeger**.
| Name | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels, queries, and Explore. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _URL_ | The URL of the Jaeger instance, e.g., `http://localhost:16686` |
| _Access_ | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. |
| _Basic Auth_ | Enable basic authentication to the Jaeger data source. |
| _User_ | User name for basic authentication. |
| _Password_ | Password for basic authentication. |
## Query traces
You can query and display traces from Jaeger via [Explore]({{< relref "../explore/index.md" >}}).
{{< docs-imagebox img="/img/docs/v70/jaeger-query-editor.png" class="docs-image--no-shadow" caption="Screenshot of the Jaeger query editor" >}}
The Jaeger query editor allows you to query by trace ID directly or selecting a trace from trace selector. To query by trace ID, insert the ID into the text input.
{{< docs-imagebox img="/img/docs/v70/jaeger-query-editor-open.png" class="docs-image--no-shadow" caption="Screenshot of the Jaeger query editor with trace selector expanded" >}}
Use the trace selector to pick particular trace from all traces logged in the time range you have selected in Explore. The trace selector has three levels of nesting:
1. The service you are interested in.
1. Particular operation is part of the selected service.
1. Specific trace in which the selected operation occurred, represented by the root operation name and trace duration.
## Linking Trace ID from logs
You can link to Jaeger trace from logs in Loki by configuring a derived field with internal link. See the [Derived fields]({{< relref "loki.md#derived-fields" >}}) section in the [Loki data source]({{< relref "loki.md" >}}) documentation for details.
docs/sources/datasources/loki.md
+208
View File
@@ -0,0 +1,208 @@
+++
title = "Loki"
description = "Guide for using Loki in Grafana"
keywords = ["grafana", "loki", "logging", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/loki"]
[menu.docs]
name = "Loki"
parent = "datasources"
weight = 800
+++
# Using Loki in Grafana
> BETA: Querying Loki data requires Grafana's Explore section.
> Grafana v6.x comes with Explore enabled by default.
> In Grafana v5.3.x and v5.4.x. you need to enable Explore manually.
> Viewing Loki data in dashboard panels is supported in Grafana v6.4+.
Grafana ships with built-in support for Loki, Grafana's log aggregation system.
Just add it as a data source and you are ready to query your log data in [Explore]({{< relref "../explore" >}}).
## Adding the data source
1. Open Grafana and make sure you are logged in.
1. In the side menu under the `Configuration` link you should find a link named `Data Sources`.
1. Click the `Add data source` button at the top.
1. Select `Loki` from the list of data sources.
> **Note:** If you're not seeing the `Data Sources` link in your side menu it means that your current user does not have the `Admin` role for the current organization.
| Name | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels, queries, and Explore. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _URL_ | The URL of the Loki instance, e.g., `http://localhost:3100` |
| _Maximum lines_ | Upper limit for number of log lines returned by Loki (default is 1000). Decrease if your browser is sluggish when displaying logs in Explore. |
### Derived fields
The Derived Fields configuration allows you to:
- Add fields parsed from the log message.
- Add a link that uses the value of the field.
You can use this functionality to link to your tracing backend directly from your logs, or link to a user profile page if a userId is present in the log line. These links appear in the [log details](/explore/#labels-and-parsed-fields).
{{< docs-imagebox img="/img/docs/v65/loki_derived_fields.png" class="docs-image--no-shadow" caption="Screenshot of the derived fields configuration" >}}
Each derived field consists of:
- **Name -** Shown in the log details as a label.
- **Regex -** A Regex pattern that runs on the log message and captures part of it as the value of the new field. Can only contain a single capture group.
- **URL/query -** If the link is external, then enter the full link URL. If the link is internal link, then this input serves as query for the target data source. In both cases, you can interpolate the value from the field with `${__value.raw }` macro.
- **Internal link -** Select if the link is internal or external. In case of internal link, a data source selector allows you to select the target data source. Only tracing data sources are supported.
You can use a debug section to see what your fields extract and how the URL is interpolated. Click **Show example log message** to show the text area where you can enter a log message.
{{< docs-imagebox img="/img/docs/v65/loki_derived_fields_debug.png" class="docs-image--no-shadow" caption="Screenshot of the derived fields debugging" >}}
The new field with the link shown in log details:
{{< docs-imagebox img="/img/docs/v65/loki_derived_fields_detail.png" class="docs-image--no-shadow" caption="Screenshot of the derived field in log detail" >}}
## Querying Logs
Querying and displaying log data from Loki is available via [Explore]({{< relref "../explore" >}}), and with the [logs panel]({{< relref "../panels/visualizations/logs-panel.md" >}}) in dashboards. Select the Loki data source, and then enter a [LogQL](https://grafana.com/docs/loki/latest/logql/) query to display your logs.
### Log Queries
A log query consists of two parts: **log stream selector**, and a **search expression**. For performance reasons you need to start by choosing a log stream by selecting a log label.
The Logs Explorer (the `Log labels` button) next to the query field shows a list of labels of available log streams. An alternative way to write a query is to use the query field's autocomplete - you start by typing a left curly brace `{` and the autocomplete menu will suggest a list of labels. Press the `Enter` key to execute the query.
Once the result is returned, the log panel shows a list of log rows and a bar chart where the x-axis shows the time and the y-axis shows the frequency/count.
<div class="medium-6 columns">
<video width="800" height="500" controls>
<source src="/assets/videos/explore_loki.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</div>
<br />
### Log Stream Selector
For the label part of the query expression, wrap it in curly braces `{}` and then use the key value syntax for selecting labels. Multiple label expressions are separated by a comma:
`{app="mysql",name="mysql-backup"}`
The following label matching operators are currently supported:
- `=` exactly equal.
- `!=` not equal.
- `=~` regex-match.
- `!~` do not regex-match.
Examples:
- `{name=~"mysql.+"}`
- `{name!~"mysql.+"}`
The [same rules that apply for Prometheus Label Selectors](https://prometheus.io/docs/prometheus/latest/querying/basics/#instant-vector-selectors) apply for Loki Log Stream Selectors.
Another way to add a label selector is in the table section. Click **Filter** beside a label to add the label to the query expression. This even works for multiple queries and will add the label selector to each query.
### Search Expression
After writing the Log Stream Selector, you can filter the results further by writing a search expression. The search expression can be just text or a regex expression.
Example queries:
- `{job="mysql"} |= "error"`
- `{name="kafka"} |~ "tsdb-ops.*io:2003"`
- `{instance=~"kafka-[23]",name="kafka"} != "kafka.server:type=ReplicaManager"`
Filter operators can be chained and will sequentially filter down the expression. The resulting log lines will satisfy every filter.
**Example**
`{job="mysql"} |= "error" != "timeout"`
The following filter types are currently supported:
- `|=` line contains string.
- `!=` line doesn't contain string.
- `|~` line matches regular expression.
- `!~` line does not match regular expression.
> **Note:** For more details about LogQL, Loki's query language, refer to the [Loki LogQL](https://grafana.com/docs/loki/latest/logql/)
## Live tailing
Loki supports Live tailing which displays logs in real-time. This feature is supported in [Explore]({{< relref "../explore/#loki-specific-features" >}}).
Note that Live Tailing relies on two Websocket connections: one between the browser and the Grafana server, and another between the Grafana server and the Loki server. If you run any reverse proxies, please configure them accordingly. The following example for Apache2 can be used for proxying between the browser and the Grafana server:
```
ProxyPassMatch "^/(api/datasources/proxy/\d+/loki/api/v1/tail)" "ws://127.0.0.1:3000/$1"
```
> **Note:** This feature is only available in Grafana v6.3+
## Log Context
When using a search expression as detailed above, you now have the ability to retrieve the context surrounding your filtered results.
By clicking the `Show Context` link on the filtered rows, you'll be able to investigate the log messages that came before and after the
log message you're interested in.
> **Note:** This feature is only available in Grafana v6.3+
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place. Variables are shown as drop-down select boxes at the top of the dashboard. These drop-down boxes make it easy to change the data being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
## Annotations
You can use any non-metric Loki query as a source for annotations. Log content will be used as annotation text and your log stream labels as tags, so there is no need for additional mapping.
> **Note:** Annotations for Loki are only available in Grafana v6.4+
## Configure the data source with provisioning
You can set up the data source via config files with Grafana's provisioning system.
You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here is an example:
```yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
jsonData:
maxLines: 1000
```
Here's another with basic auth and derived field. Keep in mind that `$` character needs to be escaped in YAML values as it is used to interpolate environment variables:
```yaml
apiVersion: 1
datasources:
- name: Loki
type: loki
access: proxy
url: http://localhost:3100
basicAuth: true
basicAuthUser: my_user
basicAuthPassword: test_password
jsonData:
maxLines: 1000
derivedFields:
# Field with internal link pointing to data source in Grafana.
# Right now, Grafana supports only Jaeger and Zipkin data sources as link targets.
# datasourceUid value can be anything, but it should be unique across all defined data source uids.
- datasourceUid: my_jaeger_uid
matcherRegex: "traceID=(\\w+)"
name: TraceID
# url will be interpreted as query for the datasource
url: "$${__value.raw}"
# Field with external link.
- matcherRegex: "traceID=(\\w+)"
name: TraceID
url: "http://localhost:16686/trace/$${__value.raw}"
```
docs/sources/datasources/mssql.md
+621
View File
@@ -0,0 +1,621 @@
+++
title = "Microsoft SQL Server"
description = "Guide for using Microsoft SQL Server in Grafana"
keywords = ["grafana", "MSSQL", "Microsoft", "SQL", "guide", "Azure SQL Database"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/mssql/"]
[menu.docs]
name = "Microsoft SQL Server"
parent = "datasources"
weight = 900
+++
# Using Microsoft SQL Server in Grafana
> Only available in Grafana v5.1+.
Grafana ships with a built-in Microsoft SQL Server (MSSQL) data source plugin that allows you to query and visualize data from any Microsoft SQL Server 2005 or newer, including Microsoft Azure SQL Database.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Configuration` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *Microsoft SQL Server* from the *Type* dropdown.
### Data source options
Name | Description
------------ | -------------
*Name* | The data source name. This is how you refer to the data source in panels and queries.
*Default* | Default data source means that it will be pre-selected for new panels.
*Host* | The IP address/hostname and optional port of your MSSQL instance. If port is omitted, default 1433 will be used.
*Database* | Name of your MSSQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Encrypt* | This option determines whether or to which extent a secure SSL TCP/IP connection will be negotiated with the server, default `false` (Grafana v5.4+).
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
### Min time interval
A lower limit for the [$__interval]({{< relref "../variables/variable-types/_index.md#the-interval-variable" >}}) and [$__interval_ms]({{< relref "../variables/variable-types/_index.md#the-interval-ms-variable" >}}) variables.
Recommended to be set to write frequency, for example `1m` if your data is written every minute.
This option can also be overridden/configured in a dashboard panel under data source options. It's important to note that this value **needs** to be formatted as a
number followed by a valid time identifier, e.g. `1m` (1 minute) or `30s` (30 seconds). The following time identifiers are supported:
Identifier | Description
------------ | -------------
`y` | year
`M` | month
`w` | week
`d` | day
`h` | hour
`m` | minute
`s` | second
`ms` | millisecond
### Database User Permissions (Important!)
The database user you specify when you add the data source should only be granted SELECT permissions on
the specified database and tables you want to query. Grafana does not validate that the query is safe. The query
could include any SQL statement. For example, statements like `DELETE FROM user;` and `DROP TABLE user;` would be
executed. To protect against this we **Highly** recommend you create a specific MSSQL user with restricted permissions.
Example:
```sql
CREATE USER grafanareader WITH PASSWORD 'password'
GRANT SELECT ON dbo.YourTable3 TO grafanareader
```
Make sure the user does not get any unwanted privileges from the public role.
### Known Issues
If you're using an older version of Microsoft SQL Server like 2008 and 2008R2 you may need to disable encryption to be able to connect.
If possible, we recommend you to use the latest service pack available for optimal compatibility.
## Query Editor
{{< docs-imagebox img="/img/docs/v51/mssql_query_editor.png" class="docs-image--no-shadow" >}}
You will find the MSSQL query editor in the metrics tab in Graph, Singlestat or Table panel's edit mode. You enter edit mode by clicking the
panel title, then edit. The editor allows you to define a SQL query to select data to be visualized.
1. Select *Format as* `Time series` (for use in Graph or Singlestat panel's among others) or `Table` (for use in Table panel among others).
1. This is the actual editor where you write your SQL queries.
1. Show help section for MSSQL below the query editor.
1. Show actual executed SQL query. Will be available first after a successful query has been executed.
1. Add an additional query where an additional query editor will be displayed.
<div class="clearfix"></div>
## Macros
To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros.
Macro example | Description
------------ | -------------
*`$__time(dateColumn)`* | Will be replaced by an expression to rename the column to *time*. For example, *dateColumn as time*
*`$__timeEpoch(dateColumn)`* | Will be replaced by an expression to convert a DATETIME column type to Unix timestamp and rename it to *time*. <br/>For example, *DATEDIFF(second, '1970-01-01', dateColumn) AS time*
*`$__timeFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name. <br/>For example, *dateColumn BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:06:17Z'*
*`$__timeFrom()`* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'*
*`$__timeTo()`* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'*
*`$__timeGroup(dateColumn,'5m'[, fillvalue])`* | Will be replaced by an expression usable in GROUP BY clause. Providing a *fillValue* of *NULL* or *floating value* will automatically fill empty series in timerange with that value. <br/>For example, *CAST(ROUND(DATEDIFF(second, '1970-01-01', time_column)/300.0, 0) as bigint)\*300*.
*`$__timeGroup(dateColumn,'5m', 0)`* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
*`$__timeGroup(dateColumn,'5m', NULL)`* | Same as above but NULL will be used as value for missing points.
*`$__timeGroup(dateColumn,'5m', previous)`* | Same as above but the previous value in that series will be used as fill value if no value has been seen yet NULL will be used (only available in Grafana 5.3+).
*`$__timeGroupAlias(dateColumn,'5m')`* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
*`$__unixEpochFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as Unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183*
*`$__unixEpochFrom()`* | Will be replaced by the start of the currently active time selection as Unix timestamp. For example, *1494410783*
*`$__unixEpochTo()`* | Will be replaced by the end of the currently active time selection as Unix timestamp. For example, *1494497183*
*`$__unixEpochNanoFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamp. For example, *dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872*
*`$__unixEpochNanoFrom()`* | Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, *1494410783152415214*
*`$__unixEpochNanoTo()`* | Will be replaced by the end of the currently active time selection as nanosecond timestamp. For example, *1494497183142514872*
*`$__unixEpochGroup(dateColumn,'5m', [fillmode])`* | Same as $__timeGroup but for times stored as Unix timestamp (only available in Grafana 5.3+).
*`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`* | Same as above but also adds a column alias (only available in Grafana 5.3+).
We plan to add many more macros. If you have suggestions for what macros you would like to see, please [open an issue](https://github.com/grafana/grafana) in our GitHub repo.
The query editor has a link named `Generated SQL` that shows up after a query has been executed, while in panel edit mode. Click on it and it will expand and show the raw interpolated SQL string that was executed.
## Table queries
If the `Format as` query option is set to `Table` then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.
**Example database table:**
```sql
CREATE TABLE [event] (
time_sec bigint,
description nvarchar(100),
tags nvarchar(100),
)
```
```sql
CREATE TABLE [mssql_types] (
c_bit bit, c_tinyint tinyint, c_smallint smallint, c_int int, c_bigint bigint, c_money money, c_smallmoney smallmoney, c_numeric numeric(10,5),
c_real real, c_decimal decimal(10,2), c_float float,
c_char char(10), c_varchar varchar(10), c_text text,
c_nchar nchar(12), c_nvarchar nvarchar(12), c_ntext ntext,
c_datetime datetime, c_datetime2 datetime2, c_smalldatetime smalldatetime, c_date date, c_time time, c_datetimeoffset datetimeoffset
)
INSERT INTO [mssql_types]
SELECT
1, 5, 20020, 980300, 1420070400, '$20000.15', '£2.15', 12345.12,
1.11, 2.22, 3.33,
'char10', 'varchar10', 'text',
N'☺nchar12☺', N'☺nvarchar12☺', N'☺text☺',
GETDATE(), CAST(GETDATE() AS DATETIME2), CAST(GETDATE() AS SMALLDATETIME), CAST(GETDATE() AS DATE), CAST(GETDATE() AS TIME), SWITCHOFFSET(CAST(GETDATE() AS DATETIMEOFFSET), '-07:00')
```
Query editor with example query:
{{< docs-imagebox img="/img/docs/v51/mssql_table_query.png" max-width="500px" class="docs-image--no-shadow" >}}
The query:
```sql
SELECT * FROM [mssql_types]
```
You can control the name of the Table panel columns by using regular `AS ` SQL column selection syntax. Example:
```sql
SELECT
c_bit as [column1], c_tinyint as [column2]
FROM
[mssql_types]
```
The resulting table panel:
{{< docs-imagebox img="/img/docs/v51/mssql_table_result.png" max-width="1489px" class="docs-image--no-shadow" >}}
## Time series queries
If you set `Format as` to `Time series`, for use in Graph panel for example, then the query must have a column named `time` that returns either a SQL datetime or any numeric datatype representing Unix epoch in seconds. You may return a column named `metric` that is used as metric name for the value column. Any column except `time` and `metric` is treated as a value column. If you omit the `metric` column, the name of the value column will be the metric name. You may select multiple value columns, each will have its name as metric.
If you return multiple value columns and a column named `metric` then this column is used as prefix for the series name (only available in Grafana 5.3+).
Resultsets of time series queries need to be sorted by time.
**Example database table:**
```sql
CREATE TABLE [event] (
time_sec bigint,
description nvarchar(100),
tags nvarchar(100),
)
```
```sql
CREATE TABLE metric_values (
time datetime,
measurement nvarchar(100),
valueOne int,
valueTwo int,
)
INSERT metric_values (time, measurement, valueOne, valueTwo) VALUES('2018-03-15 12:30:00', 'Metric A', 62, 6)
INSERT metric_values (time, measurement, valueOne, valueTwo) VALUES('2018-03-15 12:30:00', 'Metric B', 49, 11)
...
INSERT metric_values (time, measurement, valueOne, valueTwo) VALUES('2018-03-15 13:55:00', 'Metric A', 14, 25)
INSERT metric_values (time, measurement, valueOne, valueTwo) VALUES('2018-03-15 13:55:00', 'Metric B', 48, 10)
```
{{< docs-imagebox img="/img/docs/v51/mssql_time_series_one.png" class="docs-image--no-shadow docs-image--right" >}}
**Example with one `value` and one `metric` column.**
```sql
SELECT
time,
valueOne,
measurement as metric
FROM
metric_values
WHERE
$__timeFilter(time)
ORDER BY 1
```
When the above query is used in a graph panel, it will produce two series named `Metric A` and `Metric B` with the values `valueOne` and `valueTwo` plotted over `time`.
<div class="clearfix"></div>
{{< docs-imagebox img="/img/docs/v51/mssql_time_series_two.png" class="docs-image--no-shadow docs-image--right" >}}
**Example with multiple `value` columns:**
```sql
SELECT
time,
valueOne,
valueTwo
FROM
metric_values
WHERE
$__timeFilter(time)
ORDER BY 1
```
When the above query is used in a graph panel, it will produce two series named `Metric A` and `Metric B` with the values `valueOne` and `valueTwo` plotted over `time`.
<div class="clearfix"></div>
{{< docs-imagebox img="/img/docs/v51/mssql_time_series_three.png" class="docs-image--no-shadow docs-image--right" >}}
**Example using the $__timeGroup macro:**
```sql
SELECT
$__timeGroup(time, '3m') as time,
measurement as metric,
avg(valueOne)
FROM
metric_values
WHERE
$__timeFilter(time)
GROUP BY
$__timeGroup(time, '3m'),
measurement
ORDER BY 1
```
When the above query is used in a graph panel, it will produce two series named `Metric A` and `Metric B` with the values `valueOne` and `valueTwo` plotted over `time`.
Any two series lacking a value in a three-minute window will render a line between those two lines. You'll notice that the graph to the right never goes down to zero.
<div class="clearfix"></div>
{{< docs-imagebox img="/img/docs/v51/mssql_time_series_four.png" class="docs-image--no-shadow docs-image--right" >}}
**Example using the $__timeGroup macro with fill parameter set to zero:**
```sql
SELECT
$__timeGroup(time, '3m', 0) as time,
measurement as metric,
sum(valueTwo)
FROM
metric_values
WHERE
$__timeFilter(time)
GROUP BY
$__timeGroup(time, '3m'),
measurement
ORDER BY 1
```
When the above query is used in a graph panel, the result is two series named `Metric A` and `Metric B` with a sum of `valueTwo` plotted over `time`.
Any series lacking a value in a 3 minute window will have a value of zero which you'll see rendered in the graph to the right.
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
### Query Variable
If you add a template variable of the type `Query`, you can write a MSSQL query that can
return things like measurement names, key names or key values that are shown as a dropdown select box.
For example, you can have a variable that contains all values for the `hostname` column in a table if you specify a query like this in the templating variable *Query* setting.
```sql
SELECT hostname FROM host
```
A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from `hostname` and `hostname2`.
```sql
SELECT [host].[hostname], [other_host].[hostname2] FROM host JOIN other_host ON [host].[city] = [other_host].[city]
```
Another option is a query that can create a key/value variable. The query should return two columns that are named `__text` and `__value`. The `__text` column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allow you to have a friendly name as text and an id as the value. An example query with `hostname` as the text and `id` as the value:
```sql
SELECT hostname __text, id __value FROM host
```
You can also create nested variables. For example, if you had another variable named `region`. Then you could have
the hosts variable only show hosts from the current selected region with a query like this (if `region` is a multi-value variable, then use the `IN` comparison operator rather than `=` to match against multiple values):
```sql
SELECT hostname FROM host WHERE region IN ($region)
```
### Using Variables in Queries
> From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically so if it is a string value do not wrap them in quotes in where clauses.
>
> From Grafana 5.0.0, template variable values are only quoted when the template variable is a `multi-value`.
If the variable is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values.
There are two syntaxes:
`$<varname>` Example with a template variable named `hostname`:
```sql
SELECT
atimestamp time,
aint value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp
```
`[[varname]]` Example with a template variable named `hostname`:
```sql
SELECT
atimestamp as time,
aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp
```
#### Disabling Quoting for Multi-value Variables
Grafana automatically creates a quoted, comma-separated string for multi-value variables. For example: if `server01` and `server02` are selected then it will be formatted as: `'server01', 'server02'`. Do disable quoting, use the csv formatting option for variables:
`${servers:csv}`
Read more about variable formatting options in the [Variables]({{< relref "../variables/variable-types/_index.md#advanced-formatting-options" >}}) documentation.
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.
**Columns:**
Name | Description
------------ | -------------
time | The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
timeend | Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+)
text | Event description field.
tags | Optional field name to use for event tags as a comma separated string.
**Example database tables:**
```sql
CREATE TABLE [events] (
time_sec bigint,
description nvarchar(100),
tags nvarchar(100),
)
```
We also use the database table defined in [Time series queries](#time-series-queries).
**Example query using time column with epoch values:**
```sql
SELECT
time_sec as time,
description as [text],
tags
FROM
[events]
WHERE
$__unixEpochFilter(time_sec)
ORDER BY 1
```
**Example region query using time and timeend columns with epoch values:**
> Only available in Grafana v6.6+.
```sql
SELECT
time_sec as time,
time_end_sec as timeend,
description as [text],
tags
FROM
[events]
WHERE
$__unixEpochFilter(time_sec)
ORDER BY 1
```
**Example query using time column of native SQL date/time data type:**
```sql
SELECT
time,
measurement as text,
convert(varchar, valueOne) + ',' + convert(varchar, valueTwo) as tags
FROM
metric_values
WHERE
$__timeFilter(time_column)
ORDER BY 1
```
## Stored procedure support
Stored procedures have been verified to work. However, please note that we haven't done anything special to support this, so there might be edge cases where it won't work as you would expect.
Stored procedures should be supported in table, time series and annotation queries as long as you use the same naming of columns and return data in the same format as describe above under respective section.
Please note that any macro function will not work inside a stored procedure.
### Examples
{{< docs-imagebox img="/img/docs/v51/mssql_metrics_graph.png" class="docs-image--no-shadow docs-image--right" >}}
For the following examples, the database table is defined in [Time series queries](#time-series-queries). Let's say that we want to visualize four series in a graph panel, such as all combinations of columns `valueOne`, `valueTwo` and `measurement`. Graph panel to the right visualizes what we want to achieve. To solve this, we need to use two queries:
**First query:**
```sql
SELECT
$__timeGroup(time, '5m') as time,
measurement + ' - value one' as metric,
avg(valueOne) as valueOne
FROM
metric_values
WHERE
$__timeFilter(time)
GROUP BY
$__timeGroup(time, '5m'),
measurement
ORDER BY 1
```
**Second query:**
```sql
SELECT
$__timeGroup(time, '5m') as time,
measurement + ' - value two' as metric,
avg(valueTwo) as valueTwo
FROM
metric_values
GROUP BY
$__timeGroup(time, '5m'),
measurement
ORDER BY 1
```
#### Stored procedure using time in epoch format
We can define a stored procedure that will return all data we need to render 4 series in a graph panel like above.
In this case the stored procedure accepts two parameters `@from` and `@to` of `int` data types which should be a timerange (from-to) in epoch format
which will be used to filter the data to return from the stored procedure.
We're mimicking the `$__timeGroup(time, '5m')` in the select and group by expressions, and that's why there are a lot of lengthy expressions needed -
these could be extracted to MSSQL functions, if wanted.
```sql
CREATE PROCEDURE sp_test_epoch(
@from int,
@to int
) AS
BEGIN
SELECT
cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
measurement + ' - value one' as metric,
avg(valueOne) as value
FROM
metric_values
WHERE
time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
GROUP BY
cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
measurement
UNION ALL
SELECT
cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int) as time,
measurement + ' - value two' as metric,
avg(valueTwo) as value
FROM
metric_values
WHERE
time >= DATEADD(s, @from, '1970-01-01') AND time <= DATEADD(s, @to, '1970-01-01')
GROUP BY
cast(cast(DATEDIFF(second, {d '1970-01-01'}, DATEADD(second, DATEDIFF(second,GETDATE(),GETUTCDATE()), time))/600 as int)*600 as int),
measurement
ORDER BY 1
END
```
Then we can use the following query for our graph panel.
```sql
DECLARE
@from int = $__unixEpochFrom(),
@to int = $__unixEpochTo()
EXEC dbo.sp_test_epoch @from, @to
```
#### Stored procedure using time in datetime format
We can define a stored procedure that will return all data we need to render 4 series in a graph panel like above.
In this case the stored procedure accepts two parameters `@from` and `@to` of `datetime` data types which should be a timerange (from-to)
which will be used to filter the data to return from the stored procedure.
We're mimicking the `$__timeGroup(time, '5m')` in the select and group by expressions and that's why there's a lot of lengthy expressions needed -
these could be extracted to MSSQL functions, if wanted.
```sql
CREATE PROCEDURE sp_test_datetime(
@from datetime,
@to datetime
) AS
BEGIN
SELECT
cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
measurement + ' - value one' as metric,
avg(valueOne) as value
FROM
metric_values
WHERE
time >= @from AND time <= @to
GROUP BY
cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
measurement
UNION ALL
SELECT
cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int) as time,
measurement + ' - value two' as metric,
avg(valueTwo) as value
FROM
metric_values
WHERE
time >= @from AND time <= @to
GROUP BY
cast(cast(DATEDIFF(second, {d '1970-01-01'}, time)/600 as int)*600 as int),
measurement
ORDER BY 1
END
```
Then we can use the following query for our graph panel.
```sql
DECLARE
@from datetime = $__timeFrom(),
@to datetime = $__timeTo()
EXEC dbo.sp_test_datetime @from, @to
```
## Alerting
Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule
conditions.
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: MSSQL
type: mssql
url: localhost:1433
database: grafana
user: grafana
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
secureJsonData:
password: "Password!"
```
docs/sources/datasources/mysql.md
+411
View File
@@ -0,0 +1,411 @@
+++
title = "MySQL"
description = "Guide for using MySQL in Grafana"
keywords = ["grafana", "mysql", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/mysql/"]
[menu.docs]
name = "MySQL"
parent = "datasources"
weight = 1000
+++
# Using MySQL in Grafana
> Starting from Grafana v5.1 you can name the time column *time* in addition to earlier supported *time_sec*. Usage of *time_sec* will eventually be deprecated.
Grafana ships with a built-in MySQL data source plugin that allows you to query and visualize data from a MySQL compatible database.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Dashboards` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *MySQL* from the *Type* dropdown.
### Data source options
Name | Description
------------ | -------------
*Name* | The data source name. This is how you refer to the data source in panels and queries.
*Default* | Default data source means that it will be pre-selected for new panels.
*Host* | The IP address/hostname and optional port of your MySQL instance.
*Database* | Name of your MySQL database.
*User* | Database user's login/username
*Password* | Database user's password
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours. This should always be lower than configured [wait_timeout](https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_wait_timeout) in MySQL (Grafana v5.4+).
### Min time interval
A lower limit for the [$__interval]({{< relref "../variables/variable-types/_index.md#the-interval-variable" >}}) and [$__interval_ms]({{< relref "../variables/variable-types/_index.md#the-interval-ms-variable" >}}) variables.
Recommended to be set to write frequency, for example `1m` if your data is written every minute.
This option can also be overridden/configured in a dashboard panel under data source options. It's important to note that this value **needs** to be formatted as a
number followed by a valid time identifier, e.g. `1m` (1 minute) or `30s` (30 seconds). The following time identifiers are supported:
Identifier | Description
------------ | -------------
`y` | year
`M` | month
`w` | week
`d` | day
`h` | hour
`m` | minute
`s` | second
`ms` | millisecond
### Database User Permissions (Important!)
The database user you specify when you add the data source should only be granted SELECT permissions on
the specified database and tables you want to query. Grafana does not validate that the query is safe. The query
could include any SQL statement. For example, statements like `USE otherdb;` and `DROP TABLE user;` would be
executed. To protect against this we **Highly** recommend you create a specific mysql user with restricted permissions.
Example:
```sql
CREATE USER 'grafanaReader' IDENTIFIED BY 'password';
GRANT SELECT ON mydatabase.mytable TO 'grafanaReader';
```
You can use wildcards (`*`) in place of database or table if you want to grant access to more databases and tables.
## Query Editor
> Only available in Grafana v5.4+.
{{< docs-imagebox img="/img/docs/v54/mysql_query_still.png" class="docs-image--no-shadow" animated-gif="/img/docs/v54/mysql_query.gif" >}}
You find the MySQL query editor in the metrics tab in a panel's edit mode. You enter edit mode by clicking the
panel title, then edit.
The query editor has a link named `Generated SQL` that shows up after a query has been executed, while in panel edit mode. Click on it and it will expand and show the raw interpolated SQL string that was executed.
### Select table, time column and metric column (FROM)
When you enter edit mode for the first time or add a new query Grafana will try to prefill the query builder with the first table that has a timestamp column and a numeric column.
In the FROM field, Grafana will suggest tables that are in the configured database. To select a table or view in another database that your database user has access to you can manually enter a fully qualified name (database.table) like `otherDb.metrics`.
The Time column field refers to the name of the column holding your time values. Selecting a value for the Metric column field is optional. If a value is selected, the Metric column field will be used as the series name.
The metric column suggestions will only contain columns with a text datatype (text, tinytext, mediumtext, longtext, varchar, char).
If you want to use a column with a different datatype as metric column you may enter the column name with a cast: `CAST(numericColumn as CHAR)`.
You may also enter arbitrary SQL expressions in the metric column field that evaluate to a text datatype like
`CONCAT(column1, " ", CAST(numericColumn as CHAR))`.
### Columns and Aggregation functions (SELECT)
In the `SELECT` row you can specify what columns and functions you want to use.
In the column field you may write arbitrary expressions instead of a column name like `column1 * column2 / column3`.
If you use aggregate functions you need to group your resultset. The editor will automatically add a `GROUP BY time` if you add an aggregate function.
You may add further value columns by clicking the plus button and selecting `Column` from the menu. Multiple value columns will be plotted as separate series in the graph panel.
### Filter data (WHERE)
To add a filter click the plus icon to the right of the `WHERE` condition. You can remove filters by clicking on
the filter and selecting `Remove`. A filter for the current selected timerange is automatically added to new queries.
### Group By
To group by time or any other columns click the plus icon at the end of the GROUP BY row. The suggestion dropdown will only show text columns of your currently selected table but you may manually enter any column.
You can remove the group by clicking on the item and then selecting `Remove`.
If you add any grouping, all selected columns need to have an aggregate function applied. The query builder will automatically add aggregate functions to all columns without aggregate functions when you add groupings.
#### Gap Filling
Grafana can fill in missing values when you group by time. The time function accepts two arguments. The first argument is the time window that you would like to group by, and the second argument is the value you want Grafana to fill missing items with.
### Text Editor Mode (RAW)
You can switch to the raw query editor mode by clicking the hamburger icon and selecting `Switch editor mode` or by clicking `Edit SQL` below the query.
> If you use the raw query editor, be sure your query at minimum has `ORDER BY time` and a filter on the returned time range.
## Macros
To simplify syntax and to allow for dynamic parts, like date range filters, the query can contain macros.
Macro example | Description
------------ | -------------
*`$__time(dateColumn)`* | Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, *UNIX_TIMESTAMP(dateColumn) as time_sec*
*`$__timeEpoch(dateColumn)`* | Will be replaced by an expression to convert to a UNIX timestamp and rename the column to `time_sec`. For example, *UNIX_TIMESTAMP(dateColumn) as time_sec*
*`$__timeFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name. For example, *dateColumn BETWEEN FROM_UNIXTIME(1494410783) AND FROM_UNIXTIME(1494410983)*
*`$__timeFrom()`* | Will be replaced by the start of the currently active time selection. For example, *FROM_UNIXTIME(1494410783)*
*`$__timeTo()`* | Will be replaced by the end of the currently active time selection. For example, *FROM_UNIXTIME(1494410983)*
*`$__timeGroup(dateColumn,'5m')`* | Will be replaced by an expression usable in GROUP BY clause. For example, *cast(cast(UNIX_TIMESTAMP(dateColumn)/(300) as signed)*300 as signed),*
*`$__timeGroup(dateColumn,'5m', 0)`* | Same as above but with a fill parameter so missing points in that series will be added by grafana and 0 will be used as value.
*`$__timeGroup(dateColumn,'5m', NULL)`* | Same as above but NULL will be used as value for missing points.
*`$__timeGroup(dateColumn,'5m', previous)`* | Same as above but the previous value in that series will be used as fill value if no value has been seen yet NULL will be used (only available in Grafana 5.3+).
*`$__timeGroupAlias(dateColumn,'5m')`* | Will be replaced identical to $__timeGroup but with an added column alias (only available in Grafana 5.3+).
*`$__unixEpochFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as Unix timestamp. For example, *dateColumn > 1494410783 AND dateColumn < 1494497183*
*`$__unixEpochFrom()`* | Will be replaced by the start of the currently active time selection as Unix timestamp. For example, *1494410783*
*`$__unixEpochTo()`* | Will be replaced by the end of the currently active time selection as Unix timestamp. For example, *1494497183*
*`$__unixEpochNanoFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamp. For example, *dateColumn > 1494410783152415214 AND dateColumn < 1494497183142514872*
*`$__unixEpochNanoFrom()`* | Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, *1494410783152415214*
*`$__unixEpochNanoTo()`* | Will be replaced by the end of the currently active time selection as nanosecond timestamp. For example, *1494497183142514872*
*`$__unixEpochGroup(dateColumn,'5m', [fillmode])`* | Same as $__timeGroup but for times stored as Unix timestamp (only available in Grafana 5.3+).
*`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`* | Same as above but also adds a column alias (only available in Grafana 5.3+).
We plan to add many more macros. If you have suggestions for what macros you would like to see, please [open an issue](https://github.com/grafana/grafana) in our GitHub repo.
The query editor has a link named `Generated SQL` that shows up after a query has been executed, while in panel edit mode. Click on it and it will expand and show the raw interpolated SQL string that was executed.
## Table queries
If the `Format as` query option is set to `Table` then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.
Query editor with example query:
{{< docs-imagebox img="/img/docs/v45/mysql_table_query.png" >}}
The query:
```sql
SELECT
title as 'Title',
user.login as 'Created By' ,
dashboard.created as 'Created On'
FROM dashboard
INNER JOIN user on user.id = dashboard.created_by
WHERE $__timeFilter(dashboard.created)
```
You can control the name of the Table panel columns by using regular `as ` SQL column selection syntax.
The resulting table panel:
![](/img/docs/v43/mysql_table.png)
## Time series queries
If you set `Format as` to `Time series`, for use in Graph panel for example, then the query must return a column named `time` that returns either a SQL datetime or any numeric datatype representing Unix epoch.
Any column except `time` and `metric` is treated as a value column.
You may return a column named `metric` that is used as metric name for the value column.
If you return multiple value columns and a column named `metric` then this column is used as prefix for the series name (only available in Grafana 5.3+).
Resultsets of time series queries need to be sorted by time.
**Example with `metric` column:**
```sql
SELECT
$__timeGroup(time_date_time,'5m'),
min(value_double),
'min' as metric
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY time
```
**Example using the fill parameter in the $__timeGroup macro to convert null values to be zero instead:**
```sql
SELECT
$__timeGroup(createdAt,'5m',0),
sum(value_double) as value,
measurement
FROM test_data
WHERE
$__timeFilter(createdAt)
GROUP BY time, measurement
ORDER BY time
```
**Example with multiple columns:**
```sql
SELECT
$__timeGroup(time_date_time,'5m'),
min(value_double) as min_value,
max(value_double) as max_value
FROM test_data
WHERE $__timeFilter(time_date_time)
GROUP BY time
ORDER BY time
```
Currently, there is no support for a dynamic group by time based on time range and panel width.
This is something we plan to add.
## Templating
This feature is currently available in the nightly builds and will be included in the 5.0.0 release.
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different types of template variables.
### Query Variable
If you add a template variable of the type `Query`, you can write a MySQL query that can
return things like measurement names, key names or key values that are shown as a dropdown select box.
For example, you can have a variable that contains all values for the `hostname` column in a table if you specify a query like this in the templating variable *Query* setting.
```sql
SELECT hostname FROM my_host
```
A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from `hostname` and `hostname2`.
```sql
SELECT my_host.hostname, my_other_host.hostname2 FROM my_host JOIN my_other_host ON my_host.city = my_other_host.city
```
To use time range dependent macros like `$__timeFilter(column)` in your query the refresh mode of the template variable needs to be set to *On Time Range Change*.
```sql
SELECT event_name FROM event_log WHERE $__timeFilter(time_column)
```
Another option is a query that can create a key/value variable. The query should return two columns that are named `__text` and `__value`. The `__text` column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value. An example query with `hostname` as the text and `id` as the value:
```sql
SELECT hostname AS __text, id AS __value FROM my_host
```
You can also create nested variables. For example if you had another variable named `region`. Then you could have
the hosts variable only show hosts from the current selected region with a query like this (if `region` is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values):
```sql
SELECT hostname FROM my_host WHERE region IN($region)
```
#### Using `__searchFilter` to filter results in Query Variable
> Available from Grafana 6.5 and above
Using `__searchFilter` in the query field will filter the query result based on what the user types in the dropdown select box.
When nothing has been entered by the user the default value for `__searchFilter` is `%`.
> Important that you surround the `__searchFilter` expression with quotes as Grafana does not do this for you.
The example below shows how to use `__searchFilter` as part of the query field to enable searching for `hostname` while the user types in the dropdown select box.
Query
```sql
SELECT hostname FROM my_host WHERE hostname LIKE '$__searchFilter'
```
### Using Variables in Queries
From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically so if it is a string value do not wrap them in quotes in where clauses.
From Grafana 4.7.0, template variable values are only quoted when the template variable is a `multi-value`.
If the variable is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values.
There are two syntaxes:
`$<varname>` Example with a template variable named `hostname`:
```sql
SELECT
UNIX_TIMESTAMP(atimestamp) as time,
aint as value,
avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp ASC
```
`[[varname]]` Example with a template variable named `hostname`:
```sql
SELECT
UNIX_TIMESTAMP(atimestamp) as time,
aint as value,
avarchar as metric
FROM my_table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp ASC
```
#### Disabling Quoting for Multi-value Variables
Grafana automatically creates a quoted, comma-separated string for multi-value variables. For example: if `server01` and `server02` are selected then it will be formatted as: `'server01', 'server02'`. Do disable quoting, use the csv formatting option for variables:
`${servers:csv}`
Read more about variable formatting options in the [Variables]({{< relref "../variables/_index.md#advanced-formatting-options" >}}) documentation.
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.
**Example query using time column with epoch values:**
```sql
SELECT
epoch_time as time,
metric1 as text,
CONCAT(tag1, ',', tag2) as tags
FROM
public.test_data
WHERE
$__unixEpochFilter(epoch_time)
```
**Example region query using time and timeend columns with epoch values:**
> Only available in Grafana v6.6+.
```sql
SELECT
epoch_time as time,
epoch_timeend as timeend,
metric1 as text,
CONCAT(tag1, ',', tag2) as tags
FROM
public.test_data
WHERE
$__unixEpochFilter(epoch_time)
```
**Example query using time column of native SQL date/time data type:**
```sql
SELECT
native_date_time as time,
metric1 as text,
CONCAT(tag1, ',', tag2) as tags
FROM
public.test_data
WHERE
$__timeFilter(native_date_time)
```
Name | Description
------------ | -------------
time | The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
timeend | Optional name of the end date/time field. Could be a column with a native SQL date/time data type or epoch value. (Grafana v6.6+)
text | Event description field.
tags | Optional field name to use for event tags as a comma separated string.
## Alerting
Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule conditions.
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: MySQL
type: mysql
url: localhost:3306
database: grafana
user: grafana
password: password
jsonData:
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
```
docs/sources/datasources/opentsdb.md
+113
View File
@@ -0,0 +1,113 @@
+++
title = "OpenTSDB"
description = "Guide for using OpenTSDB in Grafana"
keywords = ["grafana", "opentsdb", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/opentsdb", "/docs/grafana/latest/features/datasources/opentsdb/"]
[menu.docs]
name = "OpenTSDB"
parent = "datasources"
weight = 1100
+++
# Using OpenTSDB in Grafana
Grafana ships with advanced support for OpenTSDB.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Dashboards` link you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *OpenTSDB* from the *Type* dropdown.
> **Note:** If you're not seeing the `Data Sources` link in your side menu it means that your current user does not have the `Admin` role for the current organization.
| Name | Description |
| ------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
| *Name* | The data source name. This is how you refer to the data source in panels and queries. |
| *Default* | Default data source means that it will be pre-selected for new panels. |
| *Url* | The HTTP protocol, ip and port of you opentsdb server (default port is usually 4242) |
| *Access* | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. |
| *Version* | Version = opentsdb version, either <=2.1 or 2.2 |
| *Resolution* | Metrics from opentsdb may have datapoints with either second or millisecond resolution. |
## Query editor
Open a graph in edit mode by click the title. Query editor will differ if the data source has version <=2.1 or = 2.2.
In the former version, only tags can be used to query OpenTSDB. But in the latter version, filters as well as tags
can be used to query opentsdb. Fill Policy is also introduced in OpenTSDB 2.2.
![](/img/docs/v43/opentsdb_query_editor.png)
> **Note:** While using OpenTSDB 2.2 data source, make sure you use either Filters or Tags as they are mutually exclusive. If used together, might give you weird results.
### Auto complete suggestions
As soon as you start typing metric names, tag names and tag values , you should see highlighted auto complete suggestions for them.
The autocomplete only works if the OpenTSDB suggest API is enabled.
## Templating queries
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place.
Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Query variable
Grafana's OpenTSDB data source supports template variable queries. This means you can create template variables
that fetch the values from OpenTSDB. For example, metric names, tag names, or tag values.
When using OpenTSDB with a template variable of `query` type you can use following syntax for lookup.
| Query | Description |
| --------------------------- | --------------------------------------------------------------------------------- |
| *metrics(prefix)* | Returns metric names with specific prefix (can be empty) |
| *tag_names(cpu)* | Returns tag names (i.e. keys) for a specific cpu metric |
| *tag_values(cpu, hostname)* | Returns tag values for metric cpu and tag key hostname |
| *suggest_tagk(prefix)* | Returns tag names (i.e. keys) for all metrics with specific prefix (can be empty) |
| *suggest_tagv(prefix)* | Returns tag values for all metrics with specific prefix (can be empty) |
If you do not see template variables being populated in `Preview of values` section, you need to enable
`tsd.core.meta.enable_realtime_ts` in the OpenTSDB server settings. Also, to populate metadata of
the existing time series data in OpenTSDB, you need to run `tsdb uid metasync` on the OpenTSDB server.
### Nested Templating
One template variable can be used to filter tag values for another template variable. First parameter is the metric name,
second parameter is the tag key for which you need to find tag values, and after that all other dependent template variables.
Some examples are mentioned below to make nested template queries work successfully.
| Query | Description |
| ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| *tag_values(cpu, hostname, env=$env)* | Return tag values for cpu metric, selected env tag value and tag key hostname |
| *tag_values(cpu, hostname, env=$env, region=$region)* | Return tag values for cpu metric, selected env tag value, selected region tag value and tag key hostname |
For details on OpenTSDB metric queries, check out the official [OpenTSDB documentation](http://opentsdb.net/docs/build/html/index.html)
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: OpenTsdb
type: opentsdb
access: proxy
url: http://localhost:4242
jsonData:
tsdbResolution: 1
tsdbVersion: 1
```
## Lookup limits
By default, at most 1000 records are looked up from OpenTSDB.
You can change this by modifying the "Lookup Limit" in the OpenTSDB settings page.
docs/sources/datasources/postgres.md
+423
View File
@@ -0,0 +1,423 @@
+++
title = "PostgreSQL"
description = "Guide for using PostgreSQL in Grafana"
keywords = ["grafana", "postgresql", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/postgres/"]
[menu.docs]
name = "PostgreSQL"
parent = "datasources"
weight = 1200
+++
# Using PostgreSQL in Grafana
Grafana ships with a built-in PostgreSQL data source plugin that allows you to query and visualize data from a PostgreSQL compatible database.
## Adding the data source
1. Open the side menu by clicking the Grafana icon in the top header.
1. In the side menu under the `Configuration` icon you should find a link named `Data Sources`.
1. Click the `+ Add data source` button in the top header.
1. Select *PostgreSQL* from the *Type* dropdown.
### Data source options
Name | Description
------------ | -------------
*Name* | The data source name. This is how you refer to the data source in panels and queries.
*Default* | Default data source means that it will be pre-selected for new panels.
*Host* | The IP address/hostname and optional port of your PostgreSQL instance. _Do not_ include the database name. The connection string for connecting to Postgres will not be correct and will cause errors.
*Database* | Name of your PostgreSQL database.
*User* | Database user's login/username
*Password* | Database user's password
*SSL Mode* | This option determines whether or with what priority a secure SSL TCP/IP connection will be negotiated with the server.
*Max open* | The maximum number of open connections to the database, default `unlimited` (Grafana v5.4+).
*Max idle* | The maximum number of connections in the idle connection pool, default `2` (Grafana v5.4+).
*Max lifetime* | The maximum amount of time in seconds a connection may be reused, default `14400`/4 hours (Grafana v5.4+).
*Version* | This option determines which functions are available in the query builder (only available in Grafana 5.3+).
*TimescaleDB* | TimescaleDB is a time-series database built as a PostgreSQL extension. If enabled, Grafana will use `time_bucket` in the `$__timeGroup` macro and display TimescaleDB specific aggregate functions in the query builder (only available in Grafana 5.3+).
### Min time interval
A lower limit for the [$__interval]({{< relref "../variables/variable-types/_index.md#the-interval-variable" >}}) and [$__interval_ms]({{< relref "../variables/variable-types/_index.md#the-interval-ms-variable" >}}) variables.
Recommended to be set to write frequency, for example `1m` if your data is written every minute.
This option can also be overridden/configured in a dashboard panel under data source options. It's important to note that this value **needs** to be formatted as a
number followed by a valid time identifier, e.g. `1m` (1 minute) or `30s` (30 seconds). The following time identifiers are supported:
Identifier | Description
------------ | -------------
`y` | year
`M` | month
`w` | week
`d` | day
`h` | hour
`m` | minute
`s` | second
`ms` | millisecond
### Database user permissions (Important!)
The database user you specify when you add the data source should only be granted SELECT permissions on
the specified database and tables you want to query. Grafana does not validate that the query is safe. The query
could include any SQL statement. For example, statements like `DELETE FROM user;` and `DROP TABLE user;` would be
executed. To protect against this we **highly** recommend you create a specific PostgreSQL user with restricted permissions.
Example:
```sql
CREATE USER grafanareader WITH PASSWORD 'password';
GRANT USAGE ON SCHEMA schema TO grafanareader;
GRANT SELECT ON schema.table TO grafanareader;
```
Make sure the user does not get any unwanted privileges from the public role.
## Query editor
{{< docs-imagebox img="/img/docs/v53/postgres_query_still.png" class="docs-image--no-shadow" animated-gif="/img/docs/v53/postgres_query.gif" >}}
You find the PostgreSQL query editor in the metrics tab in Graph or Singlestat panel's edit mode. You enter edit mode by clicking the
panel title, then edit.
The query editor has a link named `Generated SQL` that shows up after a query has been executed, while in panel edit mode. Click on it and it will expand and show the raw interpolated SQL string that was executed.
### Select table, time column and metric column (FROM)
When you enter edit mode for the first time or add a new query Grafana will try to prefill the query builder with the first table that has a timestamp column and a numeric column.
In the FROM field, Grafana will suggest tables that are in the `search_path` of the database user. To select a table or view not in your `search_path`
you can manually enter a fully qualified name (schema.table) like `public.metrics`.
The Time column field refers to the name of the column holding your time values. Selecting a value for the Metric column field is optional. If a value is selected, the Metric column field will be used as the series name.
The metric column suggestions will only contain columns with a text datatype (char,varchar,text).
If you want to use a column with a different datatype as metric column you may enter the column name with a cast: `ip::text`.
You may also enter arbitrary SQL expressions in the metric column field that evaluate to a text datatype like
`hostname || ' ' || container_name`.
### Columns, window, and aggregation functions (SELECT)
In the `SELECT` row you can specify what columns and functions you want to use.
In the column field you may write arbitrary expressions instead of a column name like `column1 * column2 / column3`.
The available functions in the query editor depend on the PostgreSQL version you selected when configuring the data source.
If you use aggregate functions you need to group your resultset. The editor will automatically add a `GROUP BY time` if you add an aggregate function.
The editor tries to simplify and unify this part of the query. For example:<br>
![](/img/docs/v53/postgres_select_editor.png)<br>
The above will generate the following PostgreSQL `SELECT` clause:
```sql
avg(tx_bytes) OVER (ORDER BY "time" ROWS 5 PRECEDING) AS "tx_bytes"
```
You may add further value columns by clicking the plus button and selecting `Column` from the menu. Multiple value columns will be plotted as separate series in the graph panel.
### Filter data (WHERE)
To add a filter click the plus icon to the right of the `WHERE` condition. You can remove filters by clicking on
the filter and selecting `Remove`. A filter for the current selected timerange is automatically added to new queries.
### Group by
To group by time or any other columns click the plus icon at the end of the GROUP BY row. The suggestion dropdown will only show text columns of your currently selected table but you may manually enter any column.
You can remove the group by clicking on the item and then selecting `Remove`.
If you add any grouping, all selected columns need to have an aggregate function applied. The query builder will automatically add aggregate functions to all columns without aggregate functions when you add groupings.
#### Gap filling
Grafana can fill in missing values when you group by time. The time function accepts two arguments. The first argument is the time window that you would like to group by, and the second argument is the value you want Grafana to fill missing items with.
### Text editor mode (RAW)
You can switch to the raw query editor mode by clicking the hamburger icon and selecting `Switch editor mode` or by clicking `Edit SQL` below the query.
> If you use the raw query editor, be sure your query at minimum has `ORDER BY time` and a filter on the returned time range.
## Macros
Macros can be used within a query to simplify syntax and allow for dynamic parts.
Macro example | Description
------------ | -------------
*`$__time(dateColumn)`* | Will be replaced by an expression to rename the column to `time`. For example, *dateColumn as time*
*`$__timeSec(dateColumn)`* | Will be replaced by an expression to rename the column to `time` and converting the value to Unix timestamp. For example, *extract(epoch from dateColumn) as time*
*`$__timeFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name. For example, *dateColumn BETWEEN '2017-04-21T05:01:17Z' AND '2017-04-21T05:06:17Z'*
*`$__timeFrom()`* | Will be replaced by the start of the currently active time selection. For example, *'2017-04-21T05:01:17Z'*
*`$__timeTo()`* | Will be replaced by the end of the currently active time selection. For example, *'2017-04-21T05:06:17Z'*
*`$__timeGroup(dateColumn,'5m')`* | Will be replaced by an expression usable in a GROUP BY clause. For example, *(extract(epoch from dateColumn)/300)::bigint\*300*
*`$__timeGroup(dateColumn,'5m', 0)`* | Same as above but with a fill parameter so missing points in that series will be added by Grafana and 0 will be used as the value.
*`$__timeGroup(dateColumn,'5m', NULL)`* | Same as above but NULL will be used as value for missing points.
*`$__timeGroup(dateColumn,'5m', previous)`* | Same as above but the previous value in that series will be used as fill value. If no value has been seen yet, NULL will be used (only available in Grafana 5.3+).
*`$__timeGroupAlias(dateColumn,'5m')`* | Will be replaced with an expression identical to $__timeGroup, but with an added column alias (only available in Grafana 5.3+).
*`$__unixEpochFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as Unix timestamps. For example, *dateColumn >= 1494410783 AND dateColumn <= 1494497183*
*`$__unixEpochFrom()`* | Will be replaced by the start of the currently active time selection as Unix timestamp. For example, *1494410783*
*`$__unixEpochTo()`* | Will be replaced by the end of the currently active time selection as Unix timestamp. For example, *1494497183*
*`$__unixEpochNanoFilter(dateColumn)`* | Will be replaced by a time range filter using the specified column name with times represented as nanosecond timestamps. For example, *dateColumn >= 1494410783152415214 AND dateColumn <= 1494497183142514872*
*`$__unixEpochNanoFrom()`* | Will be replaced by the start of the currently active time selection as nanosecond timestamp. For example, *1494410783152415214*
*`$__unixEpochNanoTo()`* | Will be replaced by the end of the currently active time selection as Unix timestamp. For example, *1494497183142514872*
*`$__unixEpochGroup(dateColumn,'5m', [fillmode])`* | Same as $__timeGroup, but for times stored as Unix timestamp (only available in Grafana 5.3+).
*`$__unixEpochGroupAlias(dateColumn,'5m', [fillmode])`* | Same as above, but also adds a column alias (only available in Grafana 5.3+).
We plan to add many more macros. If you have suggestions for what macros you would like to see, please [open an issue](https://github.com/grafana/grafana) in our GitHub repo.
## Table queries
If the `Format as` query option is set to `Table` then you can basically do any type of SQL query. The table panel will automatically show the results of whatever columns and rows your query returns.
Query editor with example query:
![](/img/docs/v46/postgres_table_query.png)
The query:
```sql
SELECT
title as "Title",
"user".login as "Created By",
dashboard.created as "Created On"
FROM dashboard
INNER JOIN "user" on "user".id = dashboard.created_by
WHERE $__timeFilter(dashboard.created)
```
You can control the name of the Table panel columns by using regular `as ` SQL column selection syntax.
The resulting table panel:
![postgres table](/img/docs/v46/postgres_table.png)
## Time series queries
If you set `Format as` to `Time series`, for use in Graph panel for example, then the query must return a column named `time` that returns either a SQL datetime or any numeric datatype representing Unix epoch.
Any column except `time` and `metric` are treated as a value column.
You may return a column named `metric` that is used as metric name for the value column.
If you return multiple value columns and a column named `metric` then this column is used as prefix for the series name (only available in Grafana 5.3+).
Resultsets of time series queries need to be sorted by time.
**Example with `metric` column:**
```sql
SELECT
$__timeGroup("time_date_time",'5m'),
min("value_double"),
'min' as metric
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time
ORDER BY time
```
**Example using the fill parameter in the $__timeGroup macro to convert null values to be zero instead:**
```sql
SELECT
$__timeGroup("createdAt",'5m',0),
sum(value) as value,
measurement
FROM test_data
WHERE
$__timeFilter("createdAt")
GROUP BY time, measurement
ORDER BY time
```
**Example with multiple columns:**
```sql
SELECT
$__timeGroup("time_date_time",'5m'),
min("value_double") as "min_value",
max("value_double") as "max_value"
FROM test_data
WHERE $__timeFilter("time_date_time")
GROUP BY time
ORDER BY time
```
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries you can use variables in their place. Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data being displayed in your dashboard.
Refer to [Templates and variables]({{< relref "../variables/_index.md" >}}) for an introduction to the templating feature and the different types of template variables.
### Query variable
If you add a template variable of the type `Query`, you can write a PostgreSQL query that can
return things like measurement names, key names or key values that are shown as a dropdown select box.
For example, you can have a variable that contains all values for the `hostname` column in a table if you specify a query like this in the templating variable *Query* setting.
```sql
SELECT hostname FROM host
```
A query can return multiple columns and Grafana will automatically create a list from them. For example, the query below will return a list with values from `hostname` and `hostname2`.
```sql
SELECT host.hostname, other_host.hostname2 FROM host JOIN other_host ON host.city = other_host.city
```
To use time range dependent macros like `$__timeFilter(column)` in your query the refresh mode of the template variable needs to be set to *On Time Range Change*.
```sql
SELECT event_name FROM event_log WHERE $__timeFilter(time_column)
```
Another option is a query that can create a key/value variable. The query should return two columns that are named `__text` and `__value`. The `__text` column value should be unique (if it is not unique then the first value is used). The options in the dropdown will have a text and value that allows you to have a friendly name as text and an id as the value. An example query with `hostname` as the text and `id` as the value:
```sql
SELECT hostname AS __text, id AS __value FROM host
```
You can also create nested variables. Using a variable named `region`, you could have
the hosts variable only show hosts from the current selected region with a query like this (if `region` is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values):
```sql
SELECT hostname FROM host WHERE region IN($region)
```
#### Using `__searchFilter` to filter results in Query Variable
> Available from Grafana 6.5 and above
Using `__searchFilter` in the query field will filter the query result based on what the user types in the dropdown select box.
When nothing has been entered by the user the default value for `__searchFilter` is `%`.
> Important that you surround the `__searchFilter` expression with quotes as Grafana does not do this for you.
The example below shows how to use `__searchFilter` as part of the query field to enable searching for `hostname` while the user types in the dropdown select box.
Query
```sql
SELECT hostname FROM my_host WHERE hostname LIKE '$__searchFilter'
```
### Using Variables in Queries
From Grafana 4.3.0 to 4.6.0, template variables are always quoted automatically. If your template variables are strings, do not wrap them in quotes in where clauses.
From Grafana 4.7.0, template variable values are only quoted when the template variable is a `multi-value`.
If the variable is a multi-value variable then use the `IN` comparison operator rather than `=` to match against multiple values.
There are two syntaxes:
`$<varname>` Example with a template variable named `hostname`:
```sql
SELECT
atimestamp as time,
aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in($hostname)
ORDER BY atimestamp ASC
```
`[[varname]]` Example with a template variable named `hostname`:
```sql
SELECT
atimestamp as time,
aint as value
FROM table
WHERE $__timeFilter(atimestamp) and hostname in([[hostname]])
ORDER BY atimestamp ASC
```
#### Disabling quoting for multi-value variables
Grafana automatically creates a quoted, comma-separated string for multi-value variables. For example: if `server01` and `server02` are selected then it will be formatted as: `'server01', 'server02'`. To disable quoting, use the csv formatting option for variables:
`${servers:csv}`
Read more about variable formatting options in the [Variables]({{< relref "../variables/_index.md#advanced-formatting-options" >}}) documentation.
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation queries via the Dashboard menu / Annotations view.
**Example query using time column with epoch values:**
```sql
SELECT
epoch_time as time,
metric1 as text,
concat_ws(', ', metric1::text, metric2::text) as tags
FROM
public.test_data
WHERE
$__unixEpochFilter(epoch_time)
```
**Example region query using time and timeend columns with epoch values:**
> Only available in Grafana v6.6+.
```sql
SELECT
epoch_time as time,
epoch_time_end as timeend,
metric1 as text,
concat_ws(', ', metric1::text, metric2::text) as tags
FROM
public.test_data
WHERE
$__unixEpochFilter(epoch_time)
```
**Example query using time column of native SQL date/time data type:**
```sql
SELECT
native_date_time as time,
metric1 as text,
concat_ws(', ', metric1::text, metric2::text) as tags
FROM
public.test_data
WHERE
$__timeFilter(native_date_time)
```
Name | Description
------------ | -------------
time | The name of the date/time field. Could be a column with a native SQL date/time data type or epoch value.
timeend | Optional name of the time end field, needs to be date/time data type. If set, then annotations are marked as regions between time and time-end. (Grafana v6.6+)
text | Event description field.
tags | Optional field name to use for event tags as a comma separated string.
## Alerting
Time series queries should work in alerting conditions. Table formatted queries are not yet supported in alert rule
conditions.
## Configure the data source with provisioning
It's now possible to configure data sources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source.
```yaml
apiVersion: 1
datasources:
- name: Postgres
type: postgres
url: localhost:5432
database: grafana
user: grafana
secureJsonData:
password: "Password!"
jsonData:
sslmode: "disable" # disable/require/verify-ca/verify-full
maxOpenConns: 0 # Grafana v5.4+
maxIdleConns: 2 # Grafana v5.4+
connMaxLifetime: 14400 # Grafana v5.4+
postgresVersion: 903 # 903=9.3, 904=9.4, 905=9.5, 906=9.6, 1000=10
timescaledb: false
```
If you encounter metric request errors or other issues:
- Make sure your data source YAML file parameters exactly match the example. This includes parameter names and use of quotation marks.
- Make sure the `database` name is not included in the `url`.
docs/sources/datasources/prometheus.md
+175
View File
@@ -0,0 +1,175 @@
+++
title = "Prometheus"
description = "Guide for using Prometheus in Grafana"
keywords = ["grafana", "prometheus", "guide"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/prometheus"]
[menu.docs]
name = "Prometheus"
parent = "datasources"
weight = 1300
+++
# Prometheus data source
Grafana includes built-in support for Prometheus. This topic explains options, variables, querying, and other options specific to the Prometheus data source. Refer to [Add a data source]({{< relref "add-a-data-source.md" >}}) for instructions on how to add a data source to Grafana. Only users with the organization admin role can add data sources.
## Prometheus settings
To access Prometheus settings, hover your mouse over the **Configuration** (gear) icon, then click **Data Sources**, and then click the Prometheus data source.
| Name | Description |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels and queries. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _Url_ | The URL of your Prometheus server, e.g. `http://prometheus.example.org:9090`. |
| _Access_ | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. |
| _Basic Auth_ | Enable basic authentication to the Prometheus data source. |
| _User_ | User name for basic authentication. |
| _Password_ | Password for basic authentication. |
| _Scrape interval_ | Set this to the typical scrape and evaluation interval configured in Prometheus. Defaults to 15s. |
| _Disable metrics lookup_ | Checking this option will disable the metrics chooser and metric/label support in the query field's autocomplete. This helps if you have performance issues with bigger Prometheus instances. |
| _Custom Query Parameters_ | Add custom parameters to the Prometheus query URL. For example `timeout`, `partial_response`, `dedup`, or `max_source_resolution`. Multiple parameters should be concatenated together with an '&amp;'. |
## Prometheus query editor
Below you can find information and options for Prometheus query editor in dashboard and in Explore.
### Query editor in dashboards
Open a graph in edit mode by clicking the title > Edit (or by pressing `e` key while hovering over panel).
{{< docs-imagebox img="/img/docs/v45/prometheus_query_editor_still.png"
animated-gif="/img/docs/v45/prometheus_query_editor.gif" >}}
| Name | Description |
| ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _Query expression_ | Prometheus query expression, check out the [Prometheus documentation](http://prometheus.io/docs/querying/basics/). |
| _Legend format_ | Controls the name of the time series, using name or pattern. For example `{{hostname}}` is replaced with the label value for the label `hostname`. |
| _Min step_ | An additional lower limit for the [`step` parameter of Prometheus range queries](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries) and for the `$__interval` and `$__rate_interval` variables. The limit is absolute and not modified by the _Resolution_ setting. |
| _Resolution_ | `1/1` sets both the `$__interval` variable and the [`step` parameter of Prometheus range queries](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries) such that each pixel corresponds to one data point. For better performance, lower resolutions can be picked. `1/2` only retrieves a data point for every other pixel, and `1/10` retrieves one data point per 10 pixels. Note that both _Min time interval_ and _Min step_ limit the final value of `$__interval` and `step`. |
| _Metric lookup_ | Search for metric names in this input field. |
| _Format as_ | Switch between `Table`, `Time series`, or `Heatmap`. `Table` will only work in the Table panel. `Heatmap` is suitable for displaying metrics of the Histogram type on a Heatmap panel. Under the hood, it converts cumulative histograms to regular ones and sorts series by the bucket bound. |
| _Instant_ | Perform an "instant" query, to return only the latest value that Prometheus has scraped for the requested time series. Instant queries return results much faster than normal range queries. Use them to look up label sets. |
| _Min time interval_ | This value multiplied by the denominator from the _Resolution_ setting sets a lower limit to both the `$__interval` variable and the [`step` parameter of Prometheus range queries](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries). Defaults to _Scrape interval_ as set in the data source options. |
> **Note:** Grafana modifies the request dates for queries to align them with the dynamically calculated step. This ensures consistent display of metrics data, but it can result in a small gap of data at the right edge of a graph.
#### Instant queries in dashboards
The Prometheus data source allows you to run "instant" queries, which query only the latest value.
You can visualize the results in a table panel to see all available labels of a timeseries.
Instant query results are made up only of one data point per series but can be shown in the graph panel with the help of [series overrides]({{< relref "../panels/visualizations/graph-panel.md#series-overrides" >}}).
To show them in the graph as a latest value point, add a series override and select `Points > true`.
To show a horizontal line across the whole graph, add a series override and select `Transform > constant`.
> Support for constant series overrides is available from Grafana v6.4
### Query editor in Explore
| Name | Description |
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| _Query expression_ | Prometheus query expression, check out the [Prometheus documentation](http://prometheus.io/docs/querying/basics/). |
| _Step_ | [`Step` parameter of Prometheus range queries](https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries). Time units can be used here, for example: 5s, 1m, 3h, 1d, 1y. Default unit if no unit specified is `s` (seconds). |
| _Query type_ | `Range`, `Instant`, or `Both`. When running **Range query**, the result of the query is displayed in graph and table. Instant query returns only the latest value that Prometheus has scraped for the requested time series and it is displayed in the table. When **Both** is selected, both instant query and range query is run. Result of range query is displayed in graph and the result of instant query is displayed in the table. |
## Templating
Instead of hard-coding things like server, application and sensor name in your metric queries, you can use variables in their place.
Variables are shown as dropdown select boxes at the top of the dashboard. These dropdowns make it easy to change the data
being displayed in your dashboard.
Check out the [Templating]({{< relref "../variables/_index.md" >}}) documentation for an introduction to the templating feature and the different
types of template variables.
### Query variable
Variable of the type _Query_ allows you to query Prometheus for a list of metrics, labels or label values. The Prometheus data source plugin
provides the following functions you can use in the `Query` input field.
| Name | Description |
| -------------------------------- | ----------------------------------------------------------------------- |
| _label_\__names()_ | Returns a list of label names. |
| _label_\__values(label)_ | Returns a list of label values for the `label` in every metric. |
| _label_\__values(metric, label)_ | Returns a list of label values for the `label` in the specified metric. |
| _metrics(metric)_ | Returns a list of metrics matching the specified `metric` regex. |
| _query_\__result(query)_ | Returns a list of Prometheus query result for the `query`. |
For details of what _metric names_, _label names_ and _label values_ are please refer to the [Prometheus documentation](http://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
#### Using interval and range variables
> Support for `$__range`, `$__range_s` and `$__range_ms` only available from Grafana v5.3
You can use some global built-in variables in query variables; `$__interval`, `$__interval_ms`, `$__range`, `$__range_s` and `$__range_ms`, see [Global built-in variables]({{< relref "../variables/variable-types/global-variables.md" >}}) for more information. These can be convenient to use in conjunction with the `query_result` function when you need to filter variable queries since
`label_values` function doesn't support queries.
Make sure to set the variable's `refresh` trigger to be `On Time Range Change` to get the correct instances when changing the time range on the dashboard.
**Example usage:**
Populate a variable with the busiest 5 request instances based on average QPS over the time range shown in the dashboard:
```
Query: query_result(topk(5, sum(rate(http_requests_total[$__range])) by (instance)))
Regex: /"([^"]+)"/
```
Populate a variable with the instances having a certain state over the time range shown in the dashboard, using `$__range_s`:
```
Query: query_result(max_over_time(<metric>[${__range_s}s]) != <state>)
Regex:
```
### Using `$__rate_interval` variable
> **Note:** Available in Grafana 7.2 and above
The `$__rate_interval` variable is meant to be used in the rate function. It is defined as max( `$__interval` + _Scrape interval_, 4 \* _Scrape interval_), where _Scrape interval_ is the Min step setting (AKA query_interval, a setting per PromQL query), if any is set, and otherwise the _Scrape interval_ as set in the Prometheus data source (but ignoring any Min interval setting in the panel, because the latter is modified by the resolution setting).
### Using variables in queries
There are two syntaxes:
- `$<varname>` Example: rate(http_requests_total{job=~"\$job"}[5m])
- `[[varname]]` Example: rate(http_requests_total{job=~"[[job]]"}[5m])
Why two ways? The first syntax is easier to read and write but does not allow you to use a variable in the middle of a word. When the _Multi-value_ or _Include all value_
options are enabled, Grafana converts the labels from plain text to a regex compatible string. Which means you have to use `=~` instead of `=`.
## Annotations
[Annotations]({{< relref "../dashboards/annotations.md" >}}) allow you to overlay rich event information on top of graphs. You add annotation
queries via the Dashboard menu / Annotations view.
Prometheus supports two ways to query annotations.
- A regular metric query
- A Prometheus query for pending and firing alerts (for details see [Inspecting alerts during runtime](https://prometheus.io/docs/prometheus/latest/configuration/alerting_rules/#inspecting-alerts-during-runtime))
The step option is useful to limit the number of events returned from your query.
## Get Grafana metrics into Prometheus
Grafana exposes metrics for Prometheus on the `/metrics` endpoint. We also bundle a dashboard within Grafana so you can get started viewing your metrics faster. You can import the bundled dashboard by going to the data source edit page and click the dashboard tab. There you can find a dashboard for Grafana and one for Prometheus. Import and start viewing all the metrics!
For detailed instructions, refer to [Internal Grafana metrics]({{< relref "../administration/metrics.md">}}).
## Provision the Prometheus data source
You can configure data sources using config files with Grafana's provisioning system. Read more about how it works and all the settings you can set for data sources on the [provisioning docs page]({{< relref "../administration/provisioning/#datasources" >}})
Here are some provisioning examples for this data source:
```yaml
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
access: proxy
url: http://localhost:9090
```
docs/sources/datasources/testdata.md
+54
View File
@@ -0,0 +1,54 @@
+++
title = "TestData"
keywords = ["grafana", "dashboard", "documentation", "panels", "testdata"]
type = "docs"
aliases = ["/docs/grafana/latest/features/datasources/testdata"]
[menu.docs]
name = "TestData"
parent = "datasources"
weight = 1400
+++
# Grafana TestData DB
The purpose of this data source is to make it easier to create fake data for any panel.
Using `TestData DB` you can build your own time series and have any panel render it.
This makes it much easier to verify functionality since the data can be shared very easily.
## Enable
The `TestData DB` data source is not enabled by default. To enable it:
1. In the **Configuration** menu (small gear on the left side of the screen), click **Data Sources**.
1. Click **Add Data Source**.
1. Search and click `TestData DB`.
1. Click **Save & Test** to enable it.
## Create mock data.
Once `TestData DB` is enabled, you can use it as a data source in any metric panel.
![](/img/docs/v41/test_data_add.png)
## CSV
The comma separated values scenario is the most powerful one since it lets you create any kind of graph you like.
Once you provided the numbers, `TestData DB` distributes them evenly based on the time range of your query.
![](/img/docs/v41/test_data_csv_example.png)
## Dashboards
`TestData DB` also contains some dashboards with examples.
1. Click **Configuration** > **Data Sources** > **TestData DB** > **Dashboards**.
1. **Import** the **Simple Streaming Example** dashboard.
### Commit updates to the dashboards
To submit a change to one of the current dashboards bundled with `TestData DB`, update the revision property.
Otherwise the dashboard will not be updated automatically for other Grafana users.
## Using test data in issues
If you post an issue on Github regarding time series data or rendering of time series data, we strongly advise you to use this data source to replicate the data.
That makes it much easier for the developers to replicate and solve the issue you have.
docs/sources/datasources/zipkin.md
+52
View File
@@ -0,0 +1,52 @@
+++
title = "Zipkin"
description = "Guide for using Zipkin in Grafana"
keywords = ["grafana", "zipkin", "guide", "tracing"]
type = "docs"
aliases = ["/docs/grafana/latest/datasources/zipkin"]
[menu.docs]
name = "Zipkin"
parent = "datasources"
weight = 1500
+++
# Zipkin data source
Grafana ships with built-in support for Zipkin, an open source, distributed tracing system.
Just add it as a data source and you are ready to query your traces in [Explore]({{< relref "../explore" >}}).
## Adding the data source
To access Zipkin settings, click the **Configuration** (gear) icon, then click **Data Sources**, and then click **Zipkin**.
| Name | Description |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| _Name_ | The data source name. This is how you refer to the data source in panels, queries, and Explore. |
| _Default_ | Default data source means that it will be pre-selected for new panels. |
| _URL_ | The URL of the Zipkin instance, e.g., `http://localhost:9411` |
| _Access_ | Server (default) = URL needs to be accessible from the Grafana backend/server, Browser = URL needs to be accessible from the browser. |
| _Basic Auth_ | Enable basic authentication to the Zipkin data source. |
| _User_ | User name for basic authentication. |
| _Password_ | Password for basic authentication. |
## Query traces
Querying and displaying traces from Zipkin is available via [Explore]({{< relref "../explore" >}}).
{{< docs-imagebox img="/img/docs/v70/zipkin-query-editor.png" class="docs-image--no-shadow" caption="Screenshot of the Zipkin query editor" >}}
The Zipkin query editor allows you to query by trace ID directly or selecting a trace from trace selector. To query by trace ID, insert the ID into the text input.
{{< docs-imagebox img="/img/docs/v70/zipkin-query-editor-open.png" class="docs-image--no-shadow" caption="Screenshot of the Zipkin query editor with trace selector expanded" >}}
Use the trace selector to pick particular trace from all traces logged in the time range you have selected in Explore. The trace selector has three levels of nesting:
1. The service you are interested in.
1. Particular operation is part of the selected service
1. Specific trace in which the selected operation occurred, represented by the root operation name and trace duration.
## Data mapping in the trace UI
Zipkin annotations are shown in the trace view as logs with annotation value shown under annotation key.
## Linking Trace ID from logs
You can link to Zipkin trace from logs in Loki by configuring a derived field with internal link. See [Loki documentation]({{< relref "loki#derived-fields" >}}) for details.