diff --git a/docs/sources/datasources/azure-monitor/_index.md b/docs/sources/datasources/azure-monitor/_index.md index 90452f4cc52..d66b19efdff 100644 --- a/docs/sources/datasources/azure-monitor/_index.md +++ b/docs/sources/datasources/azure-monitor/_index.md @@ -3,7 +3,6 @@ aliases: - ../data-sources/azure-monitor/ - ../features/datasources/azuremonitor/ - azuremonitor/ - - azuremonitor/deprecated-application-insights/ description: Guide for using Azure Monitor in Grafana keywords: - grafana @@ -23,6 +22,7 @@ labels: menuTitle: Azure Monitor title: Azure Monitor data source weight: 300 +last_reviewed: 2025-12-04 refs: configure-grafana-feature-toggles: - pattern: /docs/grafana/ @@ -49,6 +49,11 @@ refs: destination: /docs/grafana//dashboards/build-dashboards/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana//dashboards/build-dashboards/ + transform-data: + - pattern: /docs/grafana/ + destination: /docs/grafana//panels-visualizations/query-transform-data/transform-data/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//panels-visualizations/query-transform-data/transform-data/ configure-grafana-azure: - pattern: /docs/grafana/ destination: /docs/grafana//setup-grafana/configure-grafana/#azure @@ -63,295 +68,98 @@ refs: - pattern: /docs/grafana/ destination: /docs/grafana//setup-grafana/configure-access/configure-authentication/azuread/#enable-azure-ad-oauth-in-grafana - pattern: /docs/grafana-cloud/ - destination: /docs/grafana//setup-grafana/configure-access/configure-authentication/azuread/#enable-azure-ad-oauth-in-grafana + destination: /docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/#enable-azure-ad-oauth-in-grafana + configure-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + query-editor-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + template-variables-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + alerting-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/alerting/ + troubleshooting-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ + annotations-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/annotations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/annotations/ --- # Azure Monitor data source -Grafana ships with built-in support for Azure Monitor, the Azure service to maximize the availability and performance of applications and services in the Azure Cloud. -This topic explains configuring and querying specific to the Azure Monitor data source. +The Azure Monitor data source plugin allows you to query and visualize data from Azure Monitor, the Azure service to maximize the availability and performance of applications and services in the Azure Cloud. -For instructions on how to add a data source to Grafana, refer to the [administration documentation](ref:data-source-management). -Only users with the organization administrator role can add data sources. +## Supported Azure clouds -Once you've added the Azure Monitor data source, you can [configure it](#configure-the-data-source) so that your Grafana instance's users can create queries in its [query editor](query-editor/) when they [build dashboards](ref:build-dashboards) and use [Explore](ref:explore). +The Azure Monitor data source supports the following Azure cloud environments: -The Azure Monitor data source supports visualizing data from four Azure services: +- **Azure** - Azure public cloud (default) +- **Azure US Government** - Azure Government cloud +- **Azure China** - Azure China cloud operated by 21Vianet -- **Azure Monitor Metrics:** Collect numeric data from resources in your Azure account. -- **Azure Monitor Logs:** Collect log and performance data from your Azure account, and query using the Kusto Query Language (KQL). -- **Azure Resource Graph:** Query your Azure resources across subscriptions. -- **Azure Monitor Application Insights:** Collect trace logging data and other application performance metrics. +## Supported Azure services -## Configure the data source +The Azure Monitor data source supports the following Azure services: -**To access the data source configuration page:** +| Service | Description | +| ------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | +| **Azure Monitor Metrics** | Collect numeric data from resources in your Azure account. Supports dimensions, aggregations, and time grain configuration. | +| **Azure Monitor Logs** | Collect log and performance data from your Azure account using the Kusto Query Language (KQL). | +| **Azure Resource Graph** | Query your Azure resources across subscriptions using KQL. Useful for inventory, compliance, and resource management. | +| **Application Insights Traces** | Collect distributed trace data and correlate requests across your application components. | -1. Click **Connections** in the left-side menu. -1. Under Your connections, click **Data sources**. -1. Enter `Azure Monitor` in the search bar. -1. Click **Azure Monitor**. +## Get started - The **Settings** tab of the data source is displayed. +The following documents will help you get started with the Azure Monitor data source: -### Configure Azure Active Directory (AD) authentication +- [Configure the Azure Monitor data source](ref:configure-azure-monitor) - Set up authentication and connect to Azure +- [Azure Monitor query editor](ref:query-editor-azure-monitor) - Create and edit queries for Metrics, Logs, Traces, and Resource Graph +- [Template variables](ref:template-variables-azure-monitor) - Create dynamic dashboards with Azure Monitor variables +- [Alerting](ref:alerting-azure-monitor) - Create alert rules using Azure Monitor data +- [Troubleshooting](ref:troubleshooting-azure-monitor) - Solve common configuration and query errors -You must create an app registration and service principal in Azure AD to authenticate the data source. -For configuration details, refer to the [Azure documentation for service principals](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). +## Additional features -The app registration you create must have the `Reader` role assigned on the subscription. -For more information, refer to [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). +After you have configured the Azure Monitor data source, you can: -If you host Grafana in Azure, such as in App Service or Azure Virtual Machines, you can configure the Azure Monitor data source to use Managed Identity for secure authentication without entering credentials into Grafana. -For details, refer to [Configuring using Managed Identity](#configuring-using-managed-identity). +- Add [Annotations](ref:annotations-azure-monitor) to overlay Azure log events on your graphs. +- Configure and use [Template variables](ref:template-variables-azure-monitor) for dynamic dashboards. +- Add [Transformations](ref:transform-data) to manipulate query results. +- Set up [Alerting](ref:alerting-azure-monitor) and recording rules using Metrics, Logs, Traces, and Resource Graph queries. +- Use [Explore](ref:explore) to investigate your Azure data without building a dashboard. -You can configure the Azure Monitor data source to use Workload Identity for secure authentication without entering credentials into Grafana if you host Grafana in a Kubernetes environment, such as AKS, and require access to Azure resources. -For details, refer to [Configuring using Workload Identity](#configuring-using-workload-identity). +## Pre-built dashboards -| Name | Description | -| --------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Authentication** | Enables Managed Identity. Selecting Managed Identity hides many of the other fields. For details, see [Configuring using Managed Identity](#configuring-using-managed-identity). | -| **Azure Cloud** | Sets the national cloud for your Azure account. For most users, this is the default "Azure". For details, see the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/develop/authentication-national-cloud). | -| **Directory (tenant) ID** | Sets the directory/tenant ID for the Azure AD app registration to use for authentication. For details, see the [Azure tenant and app ID docs](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). | -| **Application (client) ID** | Sets the application/client ID for the Azure AD app registration to use for authentication. | -| **Client secret** | Sets the application client secret for the Azure AD app registration to use for authentication. For details, see the [Azure application secret docs](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret). | -| **Default subscription** | _(Optional)_ Sets a default subscription for template variables to use. | -| **Enable Basic Logs** | Allows this data source to execute queries against [Basic Logs tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) in supported Log Analytics Workspaces. These queries may incur additional costs. | +The Azure Monitor plugin includes the following pre-built dashboards: -### Provision the data source +- **Azure Monitor Overview** - Displays key metrics across your Azure subscriptions and resources. +- **Azure Storage Account** - Shows storage account metrics including availability, latency, and transactions. -You can define and configure the data source in YAML files as part of Grafana's provisioning system. -For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources). +To import a pre-built dashboard: -#### Provisioning examples +1. Go to **Connections** > **Data sources**. +1. Select your Azure Monitor data source. +1. Click the **Dashboards** tab. +1. Click **Import** next to the dashboard you want to use. -**Azure AD App Registration (client secret):** +## Related resources -```yaml -apiVersion: 1 # config file version - -datasources: - - name: Azure Monitor - type: grafana-azure-monitor-datasource - access: proxy - jsonData: - azureAuthType: clientsecret - cloudName: azuremonitor # See table below - tenantId: - clientId: - subscriptionId: # Optional, default subscription - secureJsonData: - clientSecret: - version: 1 -``` - -**Managed Identity:** - -```yaml -apiVersion: 1 # config file version - -datasources: - - name: Azure Monitor - type: grafana-azure-monitor-datasource - access: proxy - jsonData: - azureAuthType: msi - subscriptionId: # Optional, default subscription - version: 1 -``` - -**Workload Identity:** - -```yaml -apiVersion: 1 # config file version - -datasources: - - name: Azure Monitor - type: grafana-azure-monitor-datasource - access: proxy - jsonData: - azureAuthType: workloadidentity - subscriptionId: # Optional, default subscription - version: 1 -``` - -**Current User:** - -{{< admonition type="note" >}} -The `oauthPassThru` property is required for current user authentication to function. -Additionally, `disableGrafanaCache` is necessary to prevent the data source returning cached responses for resources users don't have access to. -{{< /admonition >}} - -```yaml -apiVersion: 1 # config file version - -datasources: - - name: Azure Monitor - type: grafana-azure-monitor-datasource - access: proxy - jsonData: - azureAuthType: currentuser - oauthPassThru: true - disableGrafanaCache: true - subscriptionId: # Optional, default subscription - version: 1 -``` - -#### Supported cloud names - -| Azure Cloud | `cloudName` Value | -| ------------------------------------ | -------------------------- | -| **Microsoft Azure public cloud** | `azuremonitor` (_Default_) | -| **Microsoft Chinese national cloud** | `chinaazuremonitor` | -| **US Government cloud** | `govazuremonitor` | - -{{< admonition type="note" >}} -Cloud names for current user authentication differ to the `cloudName` values in the preceding table. -The public cloud name is `AzureCloud`, the Chinese national cloud name is `AzureChinaCloud`, and the US Government cloud name is `AzureUSGovernment`. -{{< /admonition >}} - -### Configure Managed Identity - -{{< admonition type="note" >}} -Managed Identity is available only in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or Grafana OSS/Enterprise when deployed in Azure. It is not available in Grafana Cloud. -{{< /admonition >}} - -You can use managed identity to configure Azure Monitor in Grafana if you host Grafana in Azure (such as an App Service or with Azure Virtual Machines) and have managed identity enabled on your VM. -This lets you securely authenticate data sources without manually configuring credentials via Azure AD App Registrations. -For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview). - -**To enable managed identity for Grafana:** - -1. Set the `managed_identity_enabled` flag in the `[azure]` section of the [Grafana server configuration](ref:configure-grafana-azure). - - ```ini - [azure] - managed_identity_enabled = true - ``` - -2. In the Azure Monitor data source configuration, set **Authentication** to **Managed Identity**. - - This hides the directory ID, application ID, and client secret fields, and the data source uses managed identity to authenticate to Azure Monitor Metrics and Logs, and Azure Resource Graph. - - {{< figure src="/media/docs/grafana/data-sources/screenshot-managed-identity-2.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor screenshot showing Managed Identity authentication" >}} - -3. You can set the `managed_identity_client_id` field in the `[azure]` section of the [Grafana server configuration](ref:configure-grafana-azure) to allow a user-assigned managed identity to be used instead of the default system-assigned identity. - -```ini -[azure] -managed_identity_enabled = true -managed_identity_client_id = USER_ASSIGNED_IDENTITY_CLIENT_ID -``` - -### Configure Workload Identity - -You can use workload identity to configure Azure Monitor in Grafana if you host Grafana in a Kubernetes environment, such as AKS, in conjunction with managed identities. -This lets you securely authenticate data sources without manually configuring credentials via Azure AD App Registrations. -For details on workload identity, refer to the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/). - -**To enable workload identity for Grafana:** - -1. Set the `workload_identity_enabled` flag in the `[azure]` section of the [Grafana server configuration](ref:configure-grafana-azure). - - ```ini - [azure] - workload_identity_enabled = true - ``` - -2. In the Azure Monitor data source configuration, set **Authentication** to **Workload Identity**. - - This hides the directory ID, application ID, and client secret fields, and the data source uses workload identity to authenticate to Azure Monitor Metrics and Logs, and Azure Resource Graph. - - {{< figure src="/media/docs/grafana/data-sources/screenshot-workload-identity.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor screenshot showing Workload Identity authentication" >}} - -3. There are additional configuration variables that can control the authentication method.`workload_identity_tenant_id` represents the Azure AD tenant that contains the managed identity, `workload_identity_client_id` represents the client ID of the managed identity if it differs from the default client ID, `workload_identity_token_file` represents the path to the token file. Refer to the [documentation](https://azure.github.io/azure-workload-identity/docs/) for more information on what values these variables should use, if any. - - ```ini - [azure] - workload_identity_enabled = true - workload_identity_tenant_id = IDENTITY_TENANT_ID - workload_identity_client_id = IDENTITY_CLIENT_ID - workload_identity_token_file = TOKEN_FILE_PATH - ``` - -### Configure Current User authentication - -{{< admonition type="note" >}} -Current user authentication is an [experimental feature](/docs/release-life-cycle). Engineering and on-call support is not available. Documentation is either limited or not provided outside of code comments. No SLA is provided. Contact Grafana Support to enable this feature in Grafana Cloud. Aspects of Grafana may not work as expected when using this authentication method. -{{< /admonition >}} - -If your Grafana instance is configured with Azure Entra (formerly Active Directory) authentication for login, this authentication method can be used to forward the currently logged in user's credentials to the data source. The users credentials will then be used when requesting data from the data source. For details on how to configure your Grafana instance using Azure Entra refer to the [documentation](ref:configure-grafana-azure-auth). - -{{< admonition type="note" >}} -Additional configuration is required to ensure that the App Registration used to login a user via Azure provides an access token with the permissions required by the data source. - -The App Registration must be configured to issue both **Access Tokens** and **ID Tokens**. - -1. In the Azure Portal, open the App Registration that requires configuration. -2. Select **Authentication** in the side menu. -3. Under **Implicit grant and hybrid flows** check both the **Access tokens** and **ID tokens** boxes. -4. Save the changes to ensure the App Registration is updated. - -The App Registration must also be configured with additional **API Permissions** to provide authenticated users with access to the APIs utilised by the data source. - -1. In the Azure Portal, open the App Registration that requires configuration. -1. Select **API Permissions** in the side menu. -1. Ensure the `openid`, `profile`, `email`, and `offline_access` permissions are present under the **Microsoft Graph** section. If not, they must be added. -1. Select **Add a permission** and choose the following permissions. They must be added individually. Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information. - - Select **Azure Service Management** > **Delegated permissions** > `user_impersonation` > **Add permissions** - - Select **APIs my organization uses** > Search for **Log Analytics API** and select it > **Delegated permissions** > `Date.Read` > **Add permissions** - -Once all permissions have been added, the Azure authentication section in Grafana must be updated. The `scopes` section must be updated to include the `.default` scope to ensure that a token with access to all APIs declared on the App Registration is requested by Grafana. Once updated the scopes value should equal: `.default openid email profile`. -{{< /admonition >}} - -This method of authentication doesn't inherently support all backend functionality as a user's credentials won't be in scope. -Affected functionality includes alerting, reporting, and recorded queries. -In order to support backend queries when using a data source configured with current user authentication, you can configure service credentials. -Also, note that query and resource caching is disabled by default for data sources using current user authentication. - -{{< admonition type="note" >}} -To configure fallback service credentials the [feature toggle](ref:configure-grafana-feature-toggles) `idForwarding` must be set to `true` and `user_identity_fallback_credentials_enabled` must be enabled in the [Azure configuration section](ref:configure-grafana-azure) (enabled by default when `user_identity_enabled` is set to `true`). -{{< /admonition >}} - -Permissions for fallback credentials may need to be broad to appropriately support backend functionality. -For example, an alerting query created by a user is dependent on their permissions. -If a user tries to create an alert for a resource that the fallback credentials can't access, the alert will fail. - -**To enable current user authentication for Grafana:** - -1. Set the `user_identity_enabled` flag in the `[azure]` section of the [Grafana server configuration](ref:configure-grafana-azure). - By default this will also enable fallback service credentials. - If you want to disable service credentials at the instance level set `user_identity_fallback_credentials_enabled` to false. - - ```ini - [azure] - user_identity_enabled = true - ``` - -1. In the Azure Monitor data source configuration, set **Authentication** to **Current User**. - If fallback service credentials are enabled at the instance level, an additional configuration section is visible that you can use to enable or disable using service credentials for this data source. - {{< figure src="/media/docs/grafana/data-sources/screenshot-current-user.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor screenshot showing Current User authentication" >}} - -1. If you want backend functionality to work with this data source, enable service credentials and configure the data source using the most applicable credentials for your circumstances. - -## Query the data source - -The Azure Monitor data source can query data from Azure Monitor Metrics and Logs, the Azure Resource Graph, and Application Insights Traces. Each source has its own specialized query editor. - -For details, see the [query editor documentation](query-editor/). - -## Use template variables - -Instead of hard-coding details such as server, application, and sensor names in metric queries, you can use variables. -Grafana lists these variables in dropdown select boxes at the top of the dashboard to help you change the data displayed in your dashboard. -Grafana refers to such variables as template variables. - -For details, see the [template variables documentation](template-variables/). - -## Application Insights and Insights Analytics (removed) - -Until Grafana v8.0, you could query the same Azure Application Insights data using Application Insights and Insights Analytics. - -These queries were deprecated in Grafana v7.5. In Grafana v8.0, Application Insights and Insights Analytics were made read-only in favor of querying this data through Metrics and Logs. These query methods were completely removed in Grafana v9.0. - -If you're upgrading from a Grafana version prior to v9.0 and relied on Application Insights and Analytics queries, refer to the [Grafana v9.0 documentation](/docs/grafana/v9.0/datasources/azuremonitor/deprecated-application-insights/) for help migrating these queries to Metrics and Logs queries. +- [Azure Monitor documentation](https://docs.microsoft.com/en-us/azure/azure-monitor/) +- [Kusto Query Language (KQL) reference](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/) +- [Grafana community forum](https://community.grafana.com/) diff --git a/docs/sources/datasources/azure-monitor/alerting/index.md b/docs/sources/datasources/azure-monitor/alerting/index.md new file mode 100644 index 00000000000..860c1d343a4 --- /dev/null +++ b/docs/sources/datasources/azure-monitor/alerting/index.md @@ -0,0 +1,262 @@ +--- +aliases: + - ../../data-sources/azure-monitor/alerting/ +description: Set up alerts using Azure Monitor data in Grafana +keywords: + - grafana + - azure + - monitor + - alerting + - alerts + - metrics + - logs +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Alerting +title: Azure Monitor alerting +weight: 500 +refs: + alerting: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/ + alerting-fundamentals: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/fundamentals/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/fundamentals/ + create-alert-rule: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/alerting-rules/create-grafana-managed-rule/ + grafana-managed-recording-rules: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//alerting/alerting-rules/create-recording-rules/create-grafana-managed-recording-rules/ + configure-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + troubleshoot: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ +--- + +# Azure Monitor alerting + +The Azure Monitor data source supports [Grafana Alerting](ref:alerting) and [Grafana-managed recording rules](ref:grafana-managed-recording-rules), allowing you to create alert rules based on Azure metrics, logs, traces, and resource data. You can monitor your Azure environment and receive notifications when specific conditions are met. + +## Before you begin + +- Ensure you have the appropriate permissions to create alert rules in Grafana. +- Verify your Azure Monitor data source is configured and working correctly. +- Familiarize yourself with [Grafana Alerting concepts](ref:alerting-fundamentals). +- **Important**: Verify your data source uses a supported authentication method. Refer to [Authentication requirements](#authentication-requirements). + +## Supported query types for alerting + +All Azure Monitor query types support alerting and recording rules: + +| Query type | Use case | Notes | +| -------------------- | -------------------------------------------------- | -------------------------------------------------------- | +| Metrics | Threshold-based alerts on Azure resource metrics | Best suited for alerting; returns time-series data | +| Logs | Alert on log patterns, error counts, or thresholds | Use KQL to aggregate data into numeric values | +| Azure Resource Graph | Alert on resource state or configuration changes | Use count aggregations to return numeric data | +| Traces | Alert on trace data and application performance | Use aggregations to return numeric values for evaluation | + +{{< admonition type="note" >}} +Alert queries must return numeric data that Grafana can evaluate against a threshold. Queries that return only text or non-numeric data cannot be used directly for alerting. +{{< /admonition >}} + +## Authentication requirements + +Alerting and recording rules run as background processes without a user context. This means they require service-level authentication and don't work with all authentication methods. + +| Authentication method | Supported | +| -------------------------------- | ------------------------------------- | +| App Registration (client secret) | ✓ | +| Managed Identity | ✓ | +| Workload Identity | ✓ | +| Current User | ✓ (with fallback service credentials) | + +{{< admonition type="note" >}} +If you use **Current User** authentication, you must configure **fallback service credentials** for alerting and recording rules to function. User credentials aren't available for background operations, so Grafana uses the fallback credentials instead. Refer to [configure the data source](ref:configure-azure-monitor) for details on setting up fallback credentials. +{{< /admonition >}} + +## Create an alert rule + +To create an alert rule using Azure Monitor data: + +1. Go to **Alerting** > **Alert rules**. +1. Click **New alert rule**. +1. Enter a name for your alert rule. +1. In the **Define query and alert condition** section: + - Select your Azure Monitor data source. + - Configure your query (for example, a Metrics query for CPU usage or a Logs query using KQL). + - Add a **Reduce** expression if your query returns multiple series. + - Add a **Threshold** expression to define the alert condition. +1. Configure the **Set evaluation behavior**: + - Select or create a folder and evaluation group. + - Set the evaluation interval (how often the alert is checked). + - Set the pending period (how long the condition must be true before firing). +1. Add labels and annotations to provide context for notifications. +1. Click **Save rule**. + +For detailed instructions, refer to [Create a Grafana-managed alert rule](ref:create-alert-rule). + +## Example: VM CPU usage alert + +This example creates an alert that fires when virtual machine CPU usage exceeds 80%: + +1. Create a new alert rule. +1. Configure the query: + - **Service**: Metrics + - **Resource**: Select your virtual machine + - **Metric namespace**: `Microsoft.Compute/virtualMachines` + - **Metric**: `Percentage CPU` + - **Aggregation**: `Average` +1. Add expressions: + - **Reduce**: Last (to get the most recent data point) + - **Threshold**: Is above 80 +1. Set evaluation to run every 1 minute with a 5-minute pending period. +1. Save the rule. + +## Example: Error log count alert + +This example alerts when error logs exceed a threshold using a KQL query: + +1. Create a new alert rule. +1. Configure the query: + - **Service**: Logs + - **Resource**: Select your Log Analytics workspace + - **Query**: + ```kusto + AppExceptions + | where TimeGenerated > ago(5m) + | summarize ErrorCount = count() by bin(TimeGenerated, 1m) + ``` +1. Add expressions: + - **Reduce**: Max (to get the highest count in the period) + - **Threshold**: Is above 10 +1. Set evaluation to run every 5 minutes. +1. Save the rule. + +## Example: Resource count alert + +This example alerts when the number of running virtual machines drops below a threshold using Azure Resource Graph: + +1. Create a new alert rule. +1. Configure the query: + - **Service**: Azure Resource Graph + - **Subscriptions**: Select your subscriptions + - **Query**: + + ```kusto + resources + | where type == "microsoft.compute/virtualmachines" + | where properties.extended.instanceView.powerState.displayStatus == "VM running" + | summarize RunningVMs = count() + ``` + +1. Add expressions: + - **Reduce**: Last + - **Threshold**: Is below 3 +1. Set evaluation to run every 5 minutes. +1. Save the rule. + +## Best practices + +Follow these recommendations to create reliable and efficient alerts with Azure Monitor data. + +### Use appropriate query intervals + +- Set the alert evaluation interval to be greater than or equal to the minimum data resolution from Azure Monitor. +- Azure Monitor Metrics typically have 1-minute granularity at minimum. +- Avoid very short intervals (less than 1 minute) as they may cause evaluation timeouts or miss data points. + +### Reduce multiple series + +When your Azure Monitor query returns multiple time series (for example, CPU usage across multiple VMs), use the **Reduce** expression to aggregate them: + +- **Last**: Use the most recent value +- **Mean**: Average across all series +- **Max/Min**: Use the highest or lowest value +- **Sum**: Total across all series + +### Optimize Log Analytics queries + +For Logs queries used in alerting: + +- Use `summarize` to aggregate data into numeric values. +- Include appropriate time filters using `ago()` or `TimeGenerated`. +- Avoid returning large result sets; aggregate data in the query. +- Test queries in Explore before using them in alert rules. + +### Handle no data conditions + +Configure what happens when no data is returned: + +1. In the alert rule, find **Configure no data and error handling**. +1. Choose an appropriate action: + - **No Data**: Keep the alert in its current state + - **Alerting**: Treat no data as an alert condition + - **OK**: Treat no data as a healthy state + +### Test queries before alerting + +Always verify your query returns expected data before creating an alert: + +1. Go to **Explore**. +1. Select your Azure Monitor data source. +1. Run the query you plan to use for alerting. +1. Confirm the data format and values are correct. +1. Verify the query returns numeric data suitable for threshold evaluation. + +## Troubleshooting + +If your Azure Monitor alerts aren't working as expected, use the following sections to diagnose and resolve common issues. + +### Alerts not firing + +- Verify the data source uses a supported authentication method. If using Current User authentication, ensure fallback service credentials are configured. +- Check that the query returns numeric data in Explore. +- Ensure the evaluation interval allows enough time for data to be available. +- Review the alert rule's health and any error messages in the Alerting UI. + +### Authentication errors in alert evaluation + +If you see authentication errors when alerts evaluate: + +- Confirm the data source is configured with App Registration, Managed Identity, Workload Identity, or Current User with fallback service credentials. +- If using App Registration, verify the client secret hasn't expired. +- If using Current User, verify that fallback service credentials are configured and valid. +- Check that the service principal has appropriate permissions on Azure resources. + +### Query timeout errors + +- Simplify complex KQL queries. +- Reduce the time range in Log Analytics queries. +- Add more specific filters to narrow result sets. + +For additional troubleshooting help, refer to [Troubleshoot Azure Monitor](ref:troubleshoot). + +## Additional resources + +- [Grafana Alerting documentation](ref:alerting) +- [Create alert rules](ref:create-alert-rule) +- [Azure Monitor query editor](ref:query-editor) +- [Grafana-managed recording rules](ref:grafana-managed-recording-rules) diff --git a/docs/sources/datasources/azure-monitor/annotations/index.md b/docs/sources/datasources/azure-monitor/annotations/index.md new file mode 100644 index 00000000000..43fbb914a9d --- /dev/null +++ b/docs/sources/datasources/azure-monitor/annotations/index.md @@ -0,0 +1,218 @@ +--- +aliases: + - ../../data-sources/azure-monitor/annotations/ +description: Use annotations with the Azure Monitor data source in Grafana +keywords: + - grafana + - azure + - monitor + - annotations + - events + - logs +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Annotations +title: Azure Monitor annotations +weight: 450 +refs: + annotate-visualizations: + - pattern: /docs/grafana/ + destination: /docs/grafana//dashboards/build-dashboards/annotate-visualizations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//dashboards/build-dashboards/annotate-visualizations/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ +--- + +# Azure Monitor annotations + +[Annotations](ref:annotate-visualizations) overlay rich event information on top of graphs. You can use Azure Monitor Log Analytics queries to create annotations that mark important events, deployments, alerts, or other significant occurrences on your dashboards. + +## Before you begin + +- Ensure you have configured the Azure Monitor data source. +- You need access to a Log Analytics workspace containing the data you want to use for annotations. +- Annotations use Log Analytics (KQL) queries only. Metrics, Traces, and Azure Resource Graph queries are not supported for annotations. + +## Create an annotation query + +To add an Azure Monitor annotation to a dashboard: + +1. Open the dashboard where you want to add annotations. +1. Click **Dashboard settings** (gear icon) in the top navigation. +1. Select **Annotations** in the left menu. +1. Click **Add annotation query**. +1. Enter a **Name** for the annotation (e.g., "Azure Activity", "Deployments"). +1. Select your **Azure Monitor** data source. +1. Choose the **Logs** service. +1. Select a **Resource** (Log Analytics workspace or Application Insights resource). +1. Write a KQL query that returns the annotation data. +1. Click **Apply** to save. + +## Query requirements + +Your KQL query should return columns that Grafana can use to create annotations: + +| Column | Required | Description | +| ------------------ | ----------- | ------------------------------------------------------------------------------------------------ | +| `TimeGenerated` | Yes | The timestamp for the annotation. Grafana uses this to position the annotation on the time axis. | +| `Text` | Recommended | The annotation text displayed when you hover over or click the annotation. | +| Additional columns | Optional | Any other columns returned become annotation tags. | + +{{< admonition type="note" >}} +Always include a time filter in your query to limit results to the dashboard's time range. Use the `$__timeFilter()` macro. +{{< /admonition >}} + +## Annotation query examples + +The following examples demonstrate common annotation use cases. + +### Azure Activity Log events + +Display Azure Activity Log events such as resource modifications, deployments, and administrative actions: + +```kusto +AzureActivity +| where $__timeFilter(TimeGenerated) +| where Level == "Error" or Level == "Warning" or CategoryValue == "Administrative" +| project TimeGenerated, Text=OperationNameValue, Level, ResourceGroup, Caller +| order by TimeGenerated desc +| take 100 +``` + +### Deployment events + +Show deployment-related activity: + +```kusto +AzureActivity +| where $__timeFilter(TimeGenerated) +| where OperationNameValue contains "deployments" +| project TimeGenerated, Text=strcat("Deployment: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup +| order by TimeGenerated desc +``` + +### Application Insights exceptions + +Mark application exceptions as annotations: + +```kusto +AppExceptions +| where $__timeFilter(TimeGenerated) +| project TimeGenerated, Text=strcat(ProblemId, ": ", OuterMessage), SeverityLevel, AppRoleName +| order by TimeGenerated desc +| take 50 +``` + +### Custom events from Application Insights + +Display custom events logged by your application: + +```kusto +AppEvents +| where $__timeFilter(TimeGenerated) +| where Name == "DeploymentStarted" or Name == "DeploymentCompleted" +| project TimeGenerated, Text=Name, AppRoleName +| order by TimeGenerated desc +``` + +### Security alerts + +Show security-related alerts: + +```kusto +SecurityAlert +| where $__timeFilter(TimeGenerated) +| project TimeGenerated, Text=AlertName, Severity=AlertSeverity, Description +| order by TimeGenerated desc +| take 50 +``` + +### Resource health events + +Display resource health status changes: + +```kusto +AzureActivity +| where $__timeFilter(TimeGenerated) +| where CategoryValue == "ResourceHealth" +| project TimeGenerated, Text=OperationNameValue, Status=ActivityStatusValue, ResourceId +| order by TimeGenerated desc +``` + +### VM start and stop events + +Mark virtual machine state changes: + +```kusto +AzureActivity +| where $__timeFilter(TimeGenerated) +| where OperationNameValue has_any ("start", "deallocate", "restart") +| where ResourceProviderValue == "MICROSOFT.COMPUTE" +| project TimeGenerated, Text=OperationNameValue, VM=Resource, Status=ActivityStatusValue +| order by TimeGenerated desc +``` + +### Autoscale events + +Show autoscale operations: + +```kusto +AzureActivity +| where $__timeFilter(TimeGenerated) +| where OperationNameValue contains "autoscale" +| project TimeGenerated, Text=strcat("Autoscale: ", OperationNameValue), Status=ActivityStatusValue, ResourceGroup +| order by TimeGenerated desc +``` + +## Customize annotation appearance + +After creating an annotation query, you can customize its appearance: + +| Setting | Description | +| ------------- | -------------------------------------------------------------------------------------------------------- | +| **Color** | Choose a color for the annotation markers. Use different colors to distinguish between annotation types. | +| **Show in** | Select which panels display the annotations. | +| **Filter by** | Add filters to limit when annotations appear. | + +## Best practices + +Follow these recommendations when creating annotations: + +1. **Limit results**: Always use `take` or `limit` to restrict the number of annotations. Too many annotations can clutter your dashboard and impact performance. + +2. **Use time filters**: Include `$__timeFilter()` to ensure queries only return data within the dashboard's time range. + +3. **Create meaningful text**: Use `strcat()` or `project` to create descriptive annotation text that provides context at a glance. + +4. **Add relevant tags**: Include columns like `ResourceGroup`, `Severity`, or `Status` that become clickable tags for filtering. + +5. **Use descriptive names**: Name your annotations clearly (e.g., "Production Deployments", "Critical Alerts") so dashboard users understand what they represent. + +## Troubleshoot annotations + +If annotations aren't appearing as expected, try the following solutions. + +### Annotations don't appear + +- Verify the query returns data in the selected time range. +- Check that the query includes a `TimeGenerated` column. +- Test the query in the Azure Portal Log Analytics query editor. +- Ensure the annotation is enabled (toggle is on). + +### Too many annotations + +- Add more specific filters to your query. +- Use `take` to limit results. +- Narrow the time range. + +### Annotations appear at wrong times + +- Verify the `TimeGenerated` column contains the correct timestamp. +- Check your dashboard's timezone settings. diff --git a/docs/sources/datasources/azure-monitor/configure/index.md b/docs/sources/datasources/azure-monitor/configure/index.md new file mode 100644 index 00000000000..cef21b08744 --- /dev/null +++ b/docs/sources/datasources/azure-monitor/configure/index.md @@ -0,0 +1,605 @@ +--- +aliases: + - ../../data-sources/azure-monitor/configure/ +description: Guide for configuring the Azure Monitor data source in Grafana. +keywords: + - grafana + - microsoft + - azure + - monitor + - application + - insights + - log + - analytics + - guide +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Configure +title: Configure the Azure Monitor data source +weight: 200 +last_reviewed: 2025-12-04 +refs: + configure-grafana-feature-toggles: + - pattern: /docs/grafana/ + destination: /docs/grafana//setup-grafana/configure-grafana/#feature_toggles + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//setup-grafana/configure-grafana/#feature_toggles + provisioning-data-sources: + - pattern: /docs/grafana/ + destination: /docs/grafana//administration/provisioning/#data-sources + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//administration/provisioning/#data-sources + explore: + - pattern: /docs/grafana/ + destination: /docs/grafana//explore/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//explore/ + configure-grafana-azure-auth: + - pattern: /docs/grafana/ + destination: /docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/ + build-dashboards: + - pattern: /docs/grafana/ + destination: /docs/grafana//dashboards/build-dashboards/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//dashboards/build-dashboards/ + configure-grafana-azure: + - pattern: /docs/grafana/ + destination: /docs/grafana//setup-grafana/configure-grafana/#azure + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//setup-grafana/configure-grafana/#azure + data-source-management: + - pattern: /docs/grafana/ + destination: /docs/grafana//administration/data-source-management/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//administration/data-source-management/ + configure-grafana-azure-auth-scopes: + - pattern: /docs/grafana/ + destination: /docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/#enable-azure-ad-oauth-in-grafana + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//setup-grafana/configure-security/configure-authentication/azuread/#enable-azure-ad-oauth-in-grafana + data-sources: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/ + private-data-source-connect: + - pattern: /docs/grafana/ + destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + - pattern: /docs/grafana-cloud/ + destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + configure-pdc: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc +--- + +# Configure the Azure Monitor data source + +This document explains how to configure the Azure Monitor data source and the available configuration options. +For general information about data sources, refer to [Grafana data sources](ref:data-sources) and [Data source management](ref:data-source-management). + +## Before you begin + +Before configuring the Azure Monitor data source, ensure you have the following: + +- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. + Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system or [using Terraform](#configure-with-terraform). + +- **Azure prerequisites:** Depending on your chosen authentication method, you may need: + - A Microsoft Entra ID (formerly Azure AD) app registration with a service principal (for App Registration authentication) + - A Managed Identity enabled on your Azure VM or App Service (for Managed Identity authentication) + - Workload identity configured in your Kubernetes cluster (for Workload Identity authentication) + - Microsoft Entra ID authentication configured for Grafana login (for Current User authentication) + +{{< admonition type="note" >}} +**Grafana Cloud users:** Managed Identity and Workload Identity authentication methods are not available in Grafana Cloud because they require Grafana to run on your Azure infrastructure. Use **App Registration** authentication instead. +{{< /admonition >}} + +- **Azure RBAC permissions:** The identity used to authenticate must have the `Reader` role on the Azure subscription containing the resources you want to monitor. + For Log Analytics queries, the identity also needs appropriate permissions on the Log Analytics workspaces to be queried. + Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). + +{{< admonition type="note" >}} +The Azure Monitor data source plugin is built into Grafana. No additional installation is required. +{{< /admonition >}} + +## Add the data source + +To add the Azure Monitor data source: + +1. Click **Connections** in the left-side menu. +1. Click **Add new connection**. +1. Type `Azure Monitor` in the search bar. +1. Select **Azure Monitor**. +1. Click **Add new data source** in the upper right. + +You're taken to the **Settings** tab where you can configure the data source. + +## Choose an authentication method + +The Azure Monitor data source supports four authentication methods. Choose based on where Grafana is hosted and your security requirements: + +| Authentication method | Best for | Requirements | +| --------------------- | ------------------------------------------ | -------------------------------------------------------------- | +| **App Registration** | Any Grafana deployment | Microsoft Entra ID app registration with client secret | +| **Managed Identity** | Grafana hosted in Azure (VMs, App Service) | Managed identity enabled on the Azure resource | +| **Workload Identity** | Grafana in Kubernetes (AKS) | Workload identity federation configured | +| **Current User** | User-level access control | Microsoft Entra ID authentication configured for Grafana login | + +## Configure authentication + +Select one of the following authentication methods and complete the configuration. + +### App Registration + +Use a Microsoft Entra ID app registration (service principal) to authenticate. This method works with any Grafana deployment. + +#### App Registration prerequisites + +1. Create an app registration in Microsoft Entra ID. + Refer to the [Azure documentation for creating a service principal](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#get-tenant-and-app-id-values-for-signing-in). + +1. Create a client secret for the app registration. + Refer to the [Azure documentation for creating a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret). + +1. Assign the `Reader` role to the app registration on the subscription or resources you want to monitor. + Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). + +#### App Registration UI configuration + +| Setting | Description | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ | +| **Authentication** | Select **App Registration**. | +| **Azure Cloud** | The Azure environment to connect to. Select **Azure** for the public cloud, or choose Azure Government or Azure China for national clouds. | +| **Directory (tenant) ID** | The GUID that identifies your Microsoft Entra ID tenant. | +| **Application (client) ID** | The GUID for the app registration you created. | +| **Client secret** | The secret key for the app registration. Keep this secure and rotate periodically. | +| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | + +#### Provision App Registration with YAML + +```yaml +apiVersion: 1 + +datasources: + - name: Azure Monitor + type: grafana-azure-monitor-datasource + access: proxy + jsonData: + azureAuthType: clientsecret + cloudName: azuremonitor # See supported cloud names below + tenantId: + clientId: + subscriptionId: # Optional, default subscription + secureJsonData: + clientSecret: + version: 1 +``` + +### Managed Identity + +Use Azure Managed Identity for secure, credential-free authentication when Grafana is hosted in Azure. + +{{< admonition type="note" >}} +Managed Identity is available in [Azure Managed Grafana](https://azure.microsoft.com/en-us/products/managed-grafana) or self-hosted Grafana deployed in Azure. It is not available in Grafana Cloud. +{{< /admonition >}} + +#### Managed Identity prerequisites + +- Grafana must be hosted in Azure (App Service, Azure VMs, or Azure Managed Grafana). +- Managed identity must be enabled on the Azure resource hosting Grafana. +- The managed identity must have the `Reader` role on the subscription or resources you want to monitor. + +For details on Azure managed identities, refer to the [Azure documentation](https://docs.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/overview). + +#### Managed Identity Grafana server configuration + +Enable managed identity in the Grafana server configuration: + +```ini +[azure] +managed_identity_enabled = true +``` + +To use a user-assigned managed identity instead of the system-assigned identity, also set: + +```ini +[azure] +managed_identity_enabled = true +managed_identity_client_id = +``` + +Refer to [Grafana Azure configuration](ref:configure-grafana-azure) for more details. + +#### Managed Identity UI configuration + +| Setting | Description | +| ------------------------ | --------------------------------------------------------------------------------------------------- | +| **Authentication** | Select **Managed Identity**. The directory ID, application ID, and client secret fields are hidden. | +| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | + +{{< figure src="/media/docs/grafana/data-sources/screenshot-managed-identity-2.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Managed Identity" >}} + +#### Provision Managed Identity with YAML + +```yaml +apiVersion: 1 + +datasources: + - name: Azure Monitor + type: grafana-azure-monitor-datasource + access: proxy + jsonData: + azureAuthType: msi + subscriptionId: # Optional, default subscription + version: 1 +``` + +### Workload Identity + +Use Azure Workload Identity for secure authentication in Kubernetes environments like AKS. + +#### Workload Identity prerequisites + +- Grafana must be running in a Kubernetes environment with workload identity federation configured. +- The workload identity must have the `Reader` role on the subscription or resources you want to monitor. + +For details, refer to the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/). + +#### Workload Identity Grafana server configuration + +Enable workload identity in the Grafana server configuration: + +```ini +[azure] +workload_identity_enabled = true +``` + +Optional configuration variables: + +```ini +[azure] +workload_identity_enabled = true +workload_identity_tenant_id = # Microsoft Entra ID tenant containing the managed identity +workload_identity_client_id = # Client ID if different from default +workload_identity_token_file = # Path to the token file +``` + +Refer to [Grafana Azure configuration](ref:configure-grafana-azure) and the [Azure workload identity documentation](https://azure.github.io/azure-workload-identity/docs/) for more details. + +#### Workload Identity UI configuration + +| Setting | Description | +| ------------------------ | ---------------------------------------------------------------------------------------------------- | +| **Authentication** | Select **Workload Identity**. The directory ID, application ID, and client secret fields are hidden. | +| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | + +{{< figure src="/media/docs/grafana/data-sources/screenshot-workload-identity.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Workload Identity" >}} + +#### Provision Workload Identity with YAML + +```yaml +apiVersion: 1 + +datasources: + - name: Azure Monitor + type: grafana-azure-monitor-datasource + access: proxy + jsonData: + azureAuthType: workloadidentity + subscriptionId: # Optional, default subscription + version: 1 +``` + +### Current User + +Forward the logged-in Grafana user's Azure credentials to the data source for user-level access control. + +{{< admonition type="warning" >}} +Current User authentication is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. Documentation is limited. No SLA is provided. Contact Grafana Support to enable this feature in Grafana Cloud. +{{< /admonition >}} + +#### Current User prerequisites + +Your Grafana instance must be configured with Microsoft Entra ID authentication. Refer to the [Microsoft Entra ID authentication documentation](ref:configure-grafana-azure-auth). + +#### Configure your Azure App Registration + +The App Registration used for Grafana login requires additional configuration: + +**Enable token issuance:** + +1. In the Azure Portal, open your App Registration. +1. Select **Authentication** in the side menu. +1. Under **Implicit grant and hybrid flows**, check both **Access tokens** and **ID tokens**. +1. Save your changes. + +**Add API permissions:** + +1. In the Azure Portal, open your App Registration. +1. Select **API Permissions** in the side menu. +1. Ensure these permissions are present under **Microsoft Graph**: `openid`, `profile`, `email`, and `offline_access`. +1. Add the following permissions: + - **Azure Service Management** > **Delegated permissions** > `user_impersonation` + - **APIs my organization uses** > Search for **Log Analytics API** > **Delegated permissions** > `Data.Read` + +Refer to the [Azure documentation](https://learn.microsoft.com/en-us/entra/identity-platform/quickstart-configure-app-access-web-apis) for more information. + +**Update Grafana scopes:** + +Update the `scopes` section in your Grafana Azure authentication configuration to include the `.default` scope: + +``` +.default openid email profile +``` + +#### Current User Grafana server configuration + +Enable current user authentication in the Grafana server configuration: + +```ini +[azure] +user_identity_enabled = true +``` + +By default, this also enables fallback service credentials. To disable fallback credentials at the instance level: + +```ini +[azure] +user_identity_enabled = true +user_identity_fallback_credentials_enabled = false +``` + +{{< admonition type="note" >}} +To use fallback service credentials, the [feature toggle](ref:configure-grafana-feature-toggles) `idForwarding` must be set to `true`. +{{< /admonition >}} + +#### Limitations and fallback credentials + +Current User authentication doesn't support backend functionality like alerting, reporting, and recorded queries because user credentials aren't available for background operations. + +To support these features, configure **fallback service credentials**. When enabled, Grafana uses the fallback credentials for backend operations. Note that operations using fallback credentials are limited to the permissions of those credentials, not the user's permissions. + +{{< admonition type="note" >}} +Query and resource caching is disabled by default for data sources using Current User authentication. +{{< /admonition >}} + +#### Current User UI configuration + +| Setting | Description | +| -------------------------------- | ------------------------------------------------------------------------------------------- | +| **Authentication** | Select **Current User**. | +| **Default Subscription** | Click **Load Subscriptions** to populate available subscriptions, then select your default. | +| **Fallback Service Credentials** | Enable and configure credentials for backend features like alerting. | + +{{< figure src="/media/docs/grafana/data-sources/screenshot-current-user.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor data source configured with Current User authentication" >}} + +#### Provision Current User with YAML + +{{< admonition type="note" >}} +The `oauthPassThru` property is required for Current User authentication. The `disableGrafanaCache` property prevents returning cached responses for resources users don't have access to. +{{< /admonition >}} + +```yaml +apiVersion: 1 + +datasources: + - name: Azure Monitor + type: grafana-azure-monitor-datasource + access: proxy + jsonData: + azureAuthType: currentuser + oauthPassThru: true + disableGrafanaCache: true + subscriptionId: # Optional, default subscription + version: 1 +``` + +## Additional configuration options + +These settings apply to all authentication methods. + +### General settings + +| Setting | Description | +| ----------- | ------------------------------------------------------------------------------- | +| **Name** | The data source name used in panels and queries. Example: `azure-monitor-prod`. | +| **Default** | Toggle to make this the default data source for new panels. | + +### Enable Basic Logs + +Toggle **Enable Basic Logs** to allow queries against [Basic Logs tables](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/basic-logs-query?tabs=portal-1) in supported Log Analytics Workspaces. + +{{< admonition type="note" >}} +Querying Basic Logs tables incurs additional costs on a per-query basis. +{{< /admonition >}} + +### Private data source connect (Grafana Cloud only) + +If you're using Grafana Cloud and need to connect to Azure resources in a private network, use Private Data Source Connect (PDC). + +1. Click the **Private data source connect** dropdown to select your PDC configuration. +1. Click **Manage private data source connect** to view your PDC connection details. + +For more information, refer to [Private data source connect](ref:private-data-source-connect) and [Configure PDC](ref:configure-pdc). + +## Supported cloud names + +When provisioning the data source, use the following `cloudName` values: + +| Azure Cloud | `cloudName` value | +| -------------------------------- | ------------------------ | +| Microsoft Azure public cloud | `azuremonitor` (default) | +| Microsoft Chinese national cloud | `chinaazuremonitor` | +| US Government cloud | `govazuremonitor` | + +{{< admonition type="note" >}} +For Current User authentication, the cloud names differ: use `AzureCloud` for public cloud, `AzureChinaCloud` for the Chinese national cloud, and `AzureUSGovernment` for the US Government cloud. +{{< /admonition >}} + +## Verify the connection + +After configuring the data source, click **Save & test**. A successful connection displays a message confirming that the credentials are valid and have access to the configured default subscription. + +If the test fails, verify: + +- Your credentials are correct (tenant ID, client ID, client secret) +- The identity has the required Azure RBAC permissions +- For Managed Identity or Workload Identity, that the Grafana server configuration is correct +- Network connectivity to Azure endpoints + +## Provision the data source + +You can define and configure the Azure Monitor data source in YAML files as part of the Grafana provisioning system. +For more information about provisioning, refer to [Provisioning Grafana](ref:provisioning-data-sources). + +### Provision quick reference + +| Authentication method | `azureAuthType` value | Required fields | +| --------------------- | --------------------- | -------------------------------------------------- | +| App Registration | `clientsecret` | `tenantId`, `clientId`, `clientSecret` | +| Managed Identity | `msi` | None (uses VM identity) | +| Workload Identity | `workloadidentity` | None (uses pod identity) | +| Current User | `currentuser` | `oauthPassThru: true`, `disableGrafanaCache: true` | + +All methods support the optional `subscriptionId` field to set a default subscription. + +For complete YAML examples, see the [authentication method sections](#configure-authentication) above. + +## Configure with Terraform + +You can configure the Azure Monitor data source using the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). This approach enables infrastructure-as-code workflows and version control for your Grafana configuration. + +### Terraform prerequisites + +- [Terraform](https://www.terraform.io/downloads) installed +- Grafana Terraform provider configured with appropriate credentials +- For Grafana Cloud: A [Cloud Access Policy token](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) with data source permissions + +### Provider configuration + +Configure the Grafana provider to connect to your Grafana instance: + +```hcl +terraform { + required_providers { + grafana = { + source = "grafana/grafana" + version = ">= 2.0.0" + } + } +} + +# For Grafana Cloud +provider "grafana" { + url = "" + auth = "" +} + +# For self-hosted Grafana +# provider "grafana" { +# url = "http://localhost:3000" +# auth = "" +# } +``` + +### Terraform examples + +The following examples show how to configure the Azure Monitor data source for each authentication method. + +**App Registration (client secret):** + +```hcl +resource "grafana_data_source" "azure_monitor" { + type = "grafana-azure-monitor-datasource" + name = "Azure Monitor" + + json_data_encoded = jsonencode({ + azureAuthType = "clientsecret" + cloudName = "azuremonitor" + tenantId = "" + clientId = "" + subscriptionId = "" + }) + + secure_json_data_encoded = jsonencode({ + clientSecret = "" + }) +} +``` + +**Managed Identity:** + +```hcl +resource "grafana_data_source" "azure_monitor" { + type = "grafana-azure-monitor-datasource" + name = "Azure Monitor" + + json_data_encoded = jsonencode({ + azureAuthType = "msi" + subscriptionId = "" + }) +} +``` + +**Workload Identity:** + +```hcl +resource "grafana_data_source" "azure_monitor" { + type = "grafana-azure-monitor-datasource" + name = "Azure Monitor" + + json_data_encoded = jsonencode({ + azureAuthType = "workloadidentity" + subscriptionId = "" + }) +} +``` + +**Current User:** + +```hcl +resource "grafana_data_source" "azure_monitor" { + type = "grafana-azure-monitor-datasource" + name = "Azure Monitor" + + json_data_encoded = jsonencode({ + azureAuthType = "currentuser" + oauthPassThru = true + disableGrafanaCache = true + subscriptionId = "" + }) +} +``` + +**With Basic Logs enabled:** + +Add `enableBasicLogs = true` to any of the above configurations: + +```hcl +resource "grafana_data_source" "azure_monitor" { + type = "grafana-azure-monitor-datasource" + name = "Azure Monitor" + + json_data_encoded = jsonencode({ + azureAuthType = "clientsecret" + cloudName = "azuremonitor" + tenantId = "" + clientId = "" + subscriptionId = "" + enableBasicLogs = true + }) + + secure_json_data_encoded = jsonencode({ + clientSecret = "" + }) +} +``` + +For more information about the Grafana Terraform provider, refer to the [provider documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs) and the [grafana_data_source resource](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). diff --git a/docs/sources/datasources/azure-monitor/query-editor/index.md b/docs/sources/datasources/azure-monitor/query-editor/index.md index 6415be1281c..a8c763d9280 100644 --- a/docs/sources/datasources/azure-monitor/query-editor/index.md +++ b/docs/sources/datasources/azure-monitor/query-editor/index.md @@ -21,6 +21,7 @@ labels: menuTitle: Query editor title: Azure Monitor query editor weight: 300 +last_reviewed: 2025-12-04 refs: query-transform-data-query-options: - pattern: /docs/grafana/ @@ -32,30 +33,85 @@ refs: destination: /docs/grafana//panels-visualizations/query-transform-data/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana//panels-visualizations/query-transform-data/ + configure-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + explore: + - pattern: /docs/grafana/ + destination: /docs/grafana//explore/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//explore/ + troubleshoot-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/troubleshooting/ + configure-grafana-feature-toggles: + - pattern: /docs/grafana/ + destination: /docs/grafana//setup-grafana/configure-grafana/feature-toggles/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//setup-grafana/configure-grafana/feature-toggles/ + template-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + alerting-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/alerting/ + annotations-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/annotations/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/annotations/ --- # Azure Monitor query editor -This topic explains querying specific to the Azure Monitor data source. -For general documentation on querying data sources in Grafana, see [Query and transform data](ref:query-transform-data). +Grafana provides a query editor for the Azure Monitor data source, which is located on the [Explore page](ref:explore). You can also access the Azure Monitor query editor from a dashboard panel. Click the menu in the upper right of the panel and select **Edit**. -## Choose a query editing mode +This document explains querying specific to the Azure Monitor data source. +For general documentation on querying data sources in Grafana, refer to [Query and transform data](ref:query-transform-data). -The Azure Monitor data source's query editor has three modes depending on which Azure service you want to query: +The Azure Monitor data source can query data from Azure Monitor Metrics and Logs, the Azure Resource Graph, and Application Insights Traces. Each source has its own specialized query editor. + +## Before you begin + +- Ensure you have [configured the Azure Monitor data source](ref:configure-azure-monitor). +- Verify your credentials have appropriate permissions for the resources you want to query. + +## Key concepts + +If you're new to Azure Monitor, here are some key terms used throughout this documentation: + +| Term | Description | +| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **KQL (Kusto Query Language)** | The query language used for Azure Monitor Logs and Azure Resource Graph. KQL uses a pipe-based syntax similar to Unix commands and is optimized for read-only data exploration. If you know SQL, the [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet) can help you get started. | +| **Log Analytics workspace** | An Azure resource that collects and stores log data from your Azure resources, applications, and services. You query this data using KQL. | +| **Application Insights** | Azure's application performance monitoring (APM) service. It collects telemetry data like requests, exceptions, and traces from your applications. | +| **Metrics vs. Logs** | **Metrics** are lightweight numeric values collected at regular intervals (e.g., CPU percentage). **Logs** are detailed records of events with varying schemas (e.g., request logs, error messages). Metrics use a visual query builder; Logs require KQL. | + +## Choose a query editor mode + +The Azure Monitor data source's query editor has four modes depending on which Azure service you want to query: - **Metrics** for [Azure Monitor Metrics](#query-azure-monitor-metrics) - **Logs** for [Azure Monitor Logs](#query-azure-monitor-logs) -- [**Azure Resource Graph**](#query-azure-resource-graph) - **Traces** for [Application Insights Traces](#query-application-insights-traces) +- **Azure Resource Graph** for [Azure Resource Graph](#query-azure-resource-graph) ## Query Azure Monitor Metrics -Azure Monitor Metrics collects numeric data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and you can query them to investigate your resources' health and usage and maximise availability and performance. +Azure Monitor Metrics collects numeric data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and you can query them to investigate your resources' health and usage and maximize availability and performance. Monitor Metrics use a lightweight format that stores only numeric data in a specific structure and supports near real-time scenarios, making it useful for fast detection of issues. In contrast, Azure Monitor Logs can store a variety of data types, each with their own structure. -{{< figure src="/static/img/docs/azure-monitor/query-editor-metrics.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Logs Metrics sample query visualizing CPU percentage over time" >}} +{{< figure src="/static/img/docs/azure-monitor/query-editor-metrics.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Metrics sample query visualizing CPU percentage over time" >}} ### Create a Metrics query @@ -85,7 +141,7 @@ Optionally, you can apply further aggregations or filter by dimensions. The available options change depending on what is relevant to the selected metric. -You can also augment queries by using [template variables](../template-variables/). +You can also augment queries by using [template variables](ref:template-variables). ### Format legend aliases @@ -109,7 +165,7 @@ For example: | `{{ dimensionname }}` | _(Legacy for backward compatibility)_ Replaced with the name of the first dimension. | | `{{ dimensionvalue }}` | _(Legacy for backward compatibility)_ Replaced with the value of the first dimension. | -### Filter using dimensions +### Filter with dimensions Some metrics also have dimensions, which associate additional metadata. Dimensions are represented as key-value pairs assigned to each value of a metric. @@ -121,7 +177,7 @@ For more information on multi-dimensional metrics, refer to the [Azure Monitor d ## Query Azure Monitor Logs -Azure Monitor Logs collects and organises log and performance data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and makes many sources of data available to query together with the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/). +Azure Monitor Logs collects and organizes log and performance data from [supported resources](https://docs.microsoft.com/en-us/azure/azure-monitor/monitor-reference), and makes many sources of data available to query together with the [Kusto Query Language (KQL)](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/). While Azure Monitor Metrics stores only simplified numerical data, Logs can store different data types, each with their own structure. You can also perform complex analysis of Logs data by using KQL. @@ -130,6 +186,32 @@ The Azure Monitor data source also supports querying of [Basic Logs](https://lea {{< figure src="/static/img/docs/azure-monitor/query-editor-logs.png" max-width="800px" class="docs-image--no-shadow" caption="Azure Monitor Logs sample query comparing successful requests to failed requests" >}} +### Logs query builder (public preview) + +{{< admonition type="note" >}} +The Logs query builder is a [public preview feature](/docs/release-life-cycle/). It may not be enabled in all Grafana environments. +{{< /admonition >}} + +The Logs query builder provides a visual interface for building Azure Monitor Logs queries without writing KQL. This is helpful if you're new to KQL or want to quickly build simple queries. + +**To enable the Logs query builder:** + +1. Enable the `azureMonitorLogsBuilderEditor` [feature toggle](ref:configure-grafana-feature-toggles) in your Grafana configuration. +1. Restart Grafana for the change to take effect. + +**To switch between Builder and Code modes:** + +When the feature is enabled, a **Builder / Code** toggle appears in the Logs query editor: + +- **Builder**: Use the visual interface to select tables, columns, filters, and aggregations. The builder generates the KQL query for you. +- **Code**: Write KQL queries directly. Use this mode for complex queries that require full KQL capabilities. + +New queries default to Builder mode. Existing queries that were created with raw KQL remain in Code mode. + +{{< admonition type="note" >}} +You can switch from Builder to Code mode at any time to view or edit the generated KQL. However, switching from Code to Builder mode may not preserve complex queries that can't be represented in the builder interface. +{{< /admonition >}} + ### Create a Logs query **To create a Logs query:** @@ -140,13 +222,13 @@ The Azure Monitor data source also supports querying of [Basic Logs](https://lea Alternatively, you can dynamically query all resources under a single resource group or subscription. {{< admonition type="note" >}} - If a timespan is specified in the query, the overlap of the timespan between the query and the dashboard will be used as the query timespan. See the [API documentation for + If a time span is specified in the query, the overlap between the query time span and the dashboard time range will be used. See the [API documentation for details.](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters) {{< /admonition >}} 1. Enter your KQL query. -You can also augment queries by using [template variables](../template-variables/). +You can also augment queries by using [template variables](ref:template-variables). **To create a Basic Logs query:** @@ -161,7 +243,7 @@ You can also augment queries by using [template variables](../template-variables {{< /admonition >}} 1. Enter your KQL query. -You can also augment queries by using [template variables](https://grafana.com/docs/grafana//datasources/azure-monitor/template-variables/). +You can also augment queries by using [template variables](ref:template-variables). ### Logs query examples @@ -174,24 +256,28 @@ The Azure documentation includes resources to help you learn KQL: - [Tutorial: Use Kusto queries in Azure Monitor](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/tutorial?pivots=azuremonitor) - [SQL to Kusto cheat sheet](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/query/sqlcheatsheet) -> **Time-range:** The time-range that will be used for the query can be modified via the time-range switch. Selecting `Query` will only make use of time-ranges specified within the query. -> Specifying `Dashboard` will only make use of the Grafana time-range. -> If there are no time-ranges specified within the query, the default Log Analytics time-range will apply. -> For more details on this change, refer to the [Azure Monitor Logs API documentation](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters). -> If the `Intersection` option was previously chosen it will be migrated by default to `Dashboard`. +{{< admonition type="note" >}} +**Time-range:** The time-range used for the query can be modified via the time-range switch: -This example query returns a virtual machine's CPU performance, averaged over 5ms time grains: +- Selecting **Query** uses only time-ranges specified within the query. +- Selecting **Dashboard** uses only the Grafana dashboard time-range. +- If no time-range is specified in the query, the default Log Analytics time-range applies. + +For more details, refer to the [Azure Monitor Logs API documentation](https://learn.microsoft.com/en-us/rest/api/loganalytics/dataaccess/query/get?tabs=HTTP#uri-parameters). If you previously used the `Intersection` option, it has been migrated to `Dashboard`. +{{< /admonition >}} + +This example query returns a virtual machine's CPU performance, averaged over 5-minute time grains: ```kusto Perf -# $__timeFilter is a special Grafana macro that filters the results to the time span of the dashboard +// $__timeFilter is a special Grafana macro that filters the results to the time span of the dashboard | where $__timeFilter(TimeGenerated) | where CounterName == "% Processor Time" | summarize avg(CounterValue) by bin(TimeGenerated, 5m), Computer | order by TimeGenerated asc ``` -Use time series queries for values that change over time, usually for graph visualisations such as the Time series panel. +Use time series queries for values that change over time, usually for graph visualizations such as the Time series panel. Each query should return at least a datetime column and numeric value column. The result must also be sorted in ascending order by the datetime column. @@ -357,21 +443,33 @@ Application Insights stores trace data in an underlying Log Analytics workspace This query type only supports Application Insights resources. {{< /admonition >}} -Running a query of this kind will return all trace data within the timespan specified by the panel/dashboard. +1. (Optional) Specify an **Operation ID** value to filter traces. +1. (Optional) Specify **event types** to filter by. +1. (Optional) Specify **event properties** to filter by. +1. (Optional) Change the **Result format** to switch between tabular format and trace format. -Optionally, you can apply further filtering or select a specific Operation ID to query. The result format can also be switched between a tabular format or the trace format which will return the data in a format that can be used with the Trace visualization. + {{< admonition type="note" >}} + Selecting the trace format filters events to only the `trace` type. Use this format with the Trace visualization. + {{< /admonition >}} -{{< admonition type="note" >}} -Selecting the trace format will filter events with the `trace` type. -{{< /admonition >}} +Running a query returns all trace data within the time span specified by the panel or dashboard time range. -1. Specify an Operation ID value. -1. Specify event types to filter by. -1. Specify event properties to filter by. +You can also augment queries by using [template variables](ref:template-variables). -You can also augment queries by using [template variables](../template-variables/). +## Use queries for alerting and recording rules -## Working with large Azure resource data sets +All Azure Monitor query types (Metrics, Logs, Azure Resource Graph, and Traces) can be used with Grafana Alerting and recording rules. + +For detailed information about creating alert rules, supported query types, authentication requirements, and examples, refer to [Azure Monitor alerting](ref:alerting-azure-monitor). + +## Work with large Azure resource datasets If a request exceeds the [maximum allowed value of records](https://docs.microsoft.com/en-us/azure/governance/resource-graph/concepts/work-with-data#paging-results), the result is paginated and only the first page of results are returned. You can use filters to reduce the amount of records returned under that value. + +## Next steps + +- [Use template variables](../template-variables/) to create dynamic, reusable dashboards +- [Add annotations](ref:annotations-azure-monitor) to overlay events on your graphs +- [Set up alerting](ref:alerting-azure-monitor) to create alert rules based on Azure Monitor data +- [Troubleshoot](ref:troubleshoot-azure-monitor) common query and configuration issues diff --git a/docs/sources/datasources/azure-monitor/template-variables/index.md b/docs/sources/datasources/azure-monitor/template-variables/index.md index 1db472a4251..3cedadef9b5 100644 --- a/docs/sources/datasources/azure-monitor/template-variables/index.md +++ b/docs/sources/datasources/azure-monitor/template-variables/index.md @@ -23,6 +23,7 @@ labels: menuTitle: Template variables title: Azure Monitor template variables weight: 400 +last_reviewed: 2025-12-04 refs: variables: - pattern: /docs/grafana/ @@ -34,6 +35,11 @@ refs: destination: /docs/grafana//dashboards/variables/add-template-variables/ - pattern: /docs/grafana-cloud/ destination: /docs/grafana//dashboards/variables/add-template-variables/ + configure-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/configure/ --- # Azure Monitor template variables @@ -42,58 +48,173 @@ Instead of hard-coding details such as resource group or resource name values in This helps you create more interactive, dynamic, and reusable dashboards. Grafana refers to such variables as template variables. -For an introduction to templating and template variables, refer to the [Templating](ref:variables) and [Add and manage variables](ref:add-template-variables) documentation. +For an introduction to templating and template variables, refer to the [Templating](ref:variables) and [Add and manage variables](ref:add-template-variables). -## Use query variables +## Before you begin -You can specify these Azure Monitor data source queries in the Variable edit view's **Query Type** field. +- Ensure you have [configured the Azure Monitor data source](ref:configure-azure-monitor). +- If you want template variables to auto-populate subscriptions, set a **Default Subscription** in the data source configuration. -| Name | Description | -| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | -| **Subscriptions** | Returns subscriptions. | -| **Resource Groups** | Returns resource groups for a specified subscription. Supports multi-value. | -| **Namespaces** | Returns metric namespaces for the specified subscription. If a resource group is provided, only the namespaces within that group are returned. | -| **Regions** | Returns regions for the specified subscription | -| **Resource Names** | Returns a list of resource names for a specified subscription, resource group and namespace. Supports multi-value. | -| **Metric Names** | Returns a list of metric names for a resource. | -| **Workspaces** | Returns a list of workspaces for the specified subscription. | -| **Logs** | Use a KQL query to return values. | -| **Custom Namespaces** | Returns metric namespaces for the specified resource. | -| **Custom Metric Names** | Returns a list of custom metric names for the specified resource. | +## Create a template variable + +To create a template variable for Azure Monitor: + +1. Open the dashboard where you want to add the variable. +1. Click **Dashboard settings** (gear icon) in the top navigation. +1. Select **Variables** in the left menu. +1. Click **Add variable**. +1. Enter a **Name** for your variable (e.g., `subscription`, `resourceGroup`, `resource`). +1. In the **Type** dropdown, select **Query**. +1. In the **Data source** dropdown, select your Azure Monitor data source. +1. In the **Query Type** dropdown, select the appropriate query type (see [Available query types](#available-query-types)). +1. Configure any additional fields required by the selected query type. +1. Click **Run query** to preview the variable values. +1. Configure display options such as **Multi-value** or **Include All option** as needed. +1. Click **Apply** to save the variable. + +## Available query types + +The Azure Monitor data source provides the following query types for template variables: + +| Query type | Description | +| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| **Subscriptions** | Returns a list of Azure subscriptions accessible to the configured credentials. | +| **Resource Groups** | Returns resource groups for a specified subscription. Supports multi-value selection. | +| **Namespaces** | Returns metric namespaces for the specified subscription. If a resource group is specified, returns only namespaces within that group. | +| **Regions** | Returns Azure regions available for the specified subscription. | +| **Resource Names** | Returns resource names for a specified subscription, resource group, and namespace. Supports multi-value selection. | +| **Metric Names** | Returns available metric names for a specified resource. | +| **Workspaces** | Returns Log Analytics workspaces for the specified subscription. | +| **Logs** | Executes a KQL query and returns the results as variable values. See [Create a Logs variable](#create-a-logs-variable). | +| **Custom Namespaces** | Returns custom metric namespaces for a specified resource. | +| **Custom Metric Names** | Returns custom metric names for a specified resource. | {{< admonition type="note" >}} -Custom metrics cannot be emitted against a subscription or resource group. Select resources only when you need to retrieve custom metric namespaces or custom metric names associated with a specific resource. +Custom metrics cannot be emitted against a subscription or resource group. Select specific resources when retrieving custom metric namespaces or custom metric names. {{< /admonition >}} -You can use any Log Analytics Kusto Query Language (KQL) query that returns a single list of values in the `Query` field. -For example: +## Create cascading variables -| Query | List of values returned | -| ----------------------------------------------------------------------------------------- | --------------------------------------- | -| `workspace("myWorkspace").Heartbeat \| distinct Computer` | Virtual machines | -| `workspace("$workspace").Heartbeat \| distinct Computer` | Virtual machines with template variable | -| `workspace("$workspace").Perf \| distinct ObjectName` | Objects from the Perf table | -| `workspace("$workspace").Perf \| where ObjectName == "$object"` `\| distinct CounterName` | Metric names from the Perf table | +Cascading variables (also called dependent or chained variables) allow you to create dropdown menus that filter based on previous selections. This is useful for drilling down from subscription to resource group to specific resource. -### Query variable example +### Example: Subscription → Resource Group → Resource Name -This time series query uses query variables: +**Step 1: Create a Subscription variable** + +1. Create a variable named `subscription`. +1. Set **Query Type** to **Subscriptions**. + +**Step 2: Create a Resource Group variable** + +1. Create a variable named `resourceGroup`. +1. Set **Query Type** to **Resource Groups**. +1. In the **Subscription** field, select `$subscription`. + +**Step 3: Create a Resource Name variable** + +1. Create a variable named `resource`. +1. Set **Query Type** to **Resource Names**. +1. In the **Subscription** field, select `$subscription`. +1. In the **Resource Group** field, select `$resourceGroup`. +1. Select the appropriate **Namespace** for your resources (e.g., `Microsoft.Compute/virtualMachines`). + +Now when you change the subscription, the resource group dropdown updates automatically, and when you change the resource group, the resource name dropdown updates. + +## Create a Logs variable + +The **Logs** query type lets you use a KQL query to populate variable values. The query must return a single column of values. + +**To create a Logs variable:** + +1. Create a new variable with **Query Type** set to **Logs**. +1. Select a **Resource** (Log Analytics workspace or Application Insights resource). +1. Enter a KQL query that returns a single column. + +### Logs variable query examples + +| Query | Returns | +| ----------------------------------------- | ------------------------------------- | +| `Heartbeat \| distinct Computer` | List of virtual machine names | +| `Perf \| distinct ObjectName` | List of performance object names | +| `AzureActivity \| distinct ResourceGroup` | List of resource groups with activity | +| `AppRequests \| distinct Name` | List of application request names | + +You can reference other variables in your Logs query: + +```kusto +workspace("$workspace").Heartbeat | distinct Computer +``` + +```kusto +workspace("$workspace").Perf +| where ObjectName == "$object" +| distinct CounterName +``` + +## Variable refresh options + +Control when your variables refresh by setting the **Refresh** option: + +| Option | Behavior | +| ------------------------ | ----------------------------------------------------------------------------------------- | +| **On dashboard load** | Variables refresh each time the dashboard loads. Best for data that changes infrequently. | +| **On time range change** | Variables refresh when the dashboard time range changes. Use for time-sensitive queries. | + +For dashboards with many variables or complex queries, use **On dashboard load** to improve performance. + +## Use variables in queries + +After you create template variables, you can use them in your Azure Monitor queries by referencing them with the `$` prefix. + +### Metrics query example + +In a Metrics query, select your variables in the resource picker fields: + +- **Subscription**: `$subscription` +- **Resource Group**: `$resourceGroup` +- **Resource Name**: `$resource` + +### Logs query example + +Reference variables directly in your KQL queries: ```kusto Perf | where ObjectName == "$object" and CounterName == "$metric" | where TimeGenerated >= $__timeFrom() and TimeGenerated <= $__timeTo() -| where $__contains(Computer, $computer) +| where $__contains(Computer, $computer) | summarize avg(CounterValue) by bin(TimeGenerated, $__interval), Computer | order by TimeGenerated asc ``` -### Multi-value variables +## Multi-value variables -It is possible to select multiple values for **Resource Groups** and **Resource Names** and use a single metrics query pointing to those values as long as they: +You can enable **Multi-value** selection for **Resource Groups** and **Resource Names** variables. When using multi-value variables in a Metrics query, all selected resources must: -- Belong to the same subscription. -- Are in the same region. -- Are of the same type (namespace). +- Belong to the same subscription +- Be in the same Azure region +- Be of the same resource type (namespace) -Also, note that if a template variable pointing to multiple resource groups or names is used in another template variable as a parameter (e.g. to retrieve metric names), only the first value will be used. This means that the combination of the first resource group and name selected should be valid. +{{< admonition type="note" >}} +When a multi-value variable is used as a parameter in another variable query (for example, to retrieve metric names), only the first selected value is used. Ensure the first resource group and resource name combination is valid. +{{< /admonition >}} + +## Troubleshoot template variables + +If you encounter issues with template variables, try the following solutions. + +### Variable returns no values + +- Verify the Azure Monitor data source is configured correctly and can connect to Azure. +- Check that the credentials have appropriate permissions to list the requested resources. +- For cascading variables, ensure parent variables have valid selections. + +### Variable values are outdated + +- Check the **Refresh** setting and adjust if needed. +- Click the refresh icon next to the variable dropdown to manually refresh. + +### Multi-value selection not working in queries + +- Ensure the resources meet the requirements (same subscription, region, and type). +- For Logs queries, use the `$__contains()` macro to handle multi-value variables properly. diff --git a/docs/sources/datasources/azure-monitor/troubleshooting/index.md b/docs/sources/datasources/azure-monitor/troubleshooting/index.md new file mode 100644 index 00000000000..b2d5a9efc32 --- /dev/null +++ b/docs/sources/datasources/azure-monitor/troubleshooting/index.md @@ -0,0 +1,320 @@ +--- +aliases: + - ../../data-sources/azure-monitor/troubleshooting/ +description: Troubleshooting guide for the Azure Monitor data source in Grafana +keywords: + - grafana + - azure + - monitor + - troubleshooting + - errors + - authentication + - query +labels: + products: + - cloud + - enterprise + - oss +menuTitle: Troubleshoot +title: Troubleshoot Azure Monitor data source issues +weight: 500 +last_reviewed: 2025-12-04 +refs: + configure-azure-monitor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/configure/ + template-variables: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/template-variables/ + query-editor: + - pattern: /docs/grafana/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//datasources/azure-monitor/query-editor/ +--- + +# Troubleshoot Azure Monitor data source issues + +This document provides solutions to common issues you may encounter when configuring or using the Azure Monitor data source. + +## Configuration and authentication errors + +These errors typically occur when setting up the data source or when authentication credentials are invalid. + +### "Authorization failed" or "Access denied" + +**Symptoms:** + +- Save & test fails with "Authorization failed" +- Queries return "Access denied" errors +- Subscriptions don't load when clicking **Load Subscriptions** + +**Possible causes and solutions:** + +| Cause | Solution | +| -------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| App registration doesn't have required permissions | Assign the `Reader` role to the app registration on the subscription or resource group you want to monitor. Refer to the [Azure documentation for role assignments](https://docs.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current). | +| Incorrect tenant ID, client ID, or client secret | Verify the credentials in the Azure Portal under **App registrations** > your app > **Overview** (for IDs) and **Certificates & secrets** (for secret). | +| Client secret has expired | Create a new client secret in Azure and update the data source configuration. | +| Managed Identity not enabled on the Azure resource | For VMs, enable managed identity in the Azure Portal under **Identity**. For App Service, enable it under **Identity** in the app settings. | +| Managed Identity not assigned the Reader role | Assign the `Reader` role to the managed identity on the target subscription or resources. | + +### "Invalid client secret" or "Client secret not found" + +**Symptoms:** + +- Authentication fails immediately after configuration +- Error message references invalid credentials + +**Solutions:** + +1. Ensure you copied the client secret **value**, not the secret ID. In Azure Portal under **Certificates & secrets**, the secret value is only shown once when created. The secret ID is a different identifier and won't work for authentication. +2. Verify the client secret was copied correctly (no extra spaces or truncation). +3. Check if the secret has expired in Azure Portal under **App registrations** > your app > **Certificates & secrets**. +4. Create a new secret and update the data source configuration. + +### "Tenant not found" or "Invalid tenant ID" + +**Symptoms:** + +- Data source test fails with tenant-related errors +- Unable to authenticate + +**Solutions:** + +1. Verify the Directory (tenant) ID in Azure Portal under **Microsoft Entra ID** > **Overview**. +2. Ensure you're using the correct Azure cloud setting (Azure, Azure Government, or Azure China). +3. Check that the tenant ID is a valid GUID format. + +### Managed Identity not working + +**Symptoms:** + +- Managed Identity option is available but authentication fails +- Error: "Managed identity authentication is not available" + +**Solutions:** + +1. Verify `managed_identity_enabled = true` is set in the Grafana server configuration under `[azure]`. +2. Confirm the Azure resource hosting Grafana has managed identity enabled. +3. For user-assigned managed identity, ensure `managed_identity_client_id` is set correctly. +4. Verify the managed identity has the `Reader` role on the target resources. +5. Restart Grafana after changing server configuration. + +### Workload Identity not working + +**Symptoms:** + +- Workload Identity authentication fails in Kubernetes/AKS environment +- Token file errors + +**Solutions:** + +1. Verify `workload_identity_enabled = true` is set in the Grafana server configuration. +2. Check that the service account is correctly annotated for workload identity. +3. Verify the federated credential is configured in Azure. +4. Ensure the token path is accessible to the Grafana pod. +5. Check the workload identity webhook is running in the cluster. + +## Query errors + +These errors occur when executing queries against Azure Monitor services. + +### "No data" or empty results + +**Symptoms:** + +- Query executes without error but returns no data +- Charts show "No data" message + +**Possible causes and solutions:** + +| Cause | Solution | +| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | +| Time range doesn't contain data | Expand the dashboard time range or verify data exists in Azure Portal. | +| Wrong resource selected | Verify you've selected the correct subscription, resource group, and resource. | +| Metric not available for resource | Not all metrics are available for all resources. Check available metrics in Azure Portal under the resource's **Metrics** blade. | +| Metric has no values | Some metrics only populate under certain conditions (e.g., error counts when errors occur). | +| Permissions issue | Verify the identity has read access to the specific resource. | + +### "Bad request" or "Invalid query" + +**Symptoms:** + +- Query fails with 400 error +- Error message indicates query syntax issues + +**Solutions for Logs queries:** + +1. Validate your KQL syntax in the Azure Portal Log Analytics query editor. +2. Check for typos in table names or column names. +3. Ensure referenced tables exist in the selected workspace. +4. Verify the time range is valid (not in the future, not too far in the past for data retention). + +**Solutions for Metrics queries:** + +1. Verify the metric name is valid for the selected resource type. +2. Check that dimension filters use valid dimension names and values. +3. Ensure the aggregation type is supported for the selected metric. + +### "Resource not found" + +**Symptoms:** + +- Query fails with 404 error +- Resource picker shows resources that can't be queried + +**Solutions:** + +1. Verify the resource still exists in Azure (it may have been deleted or moved). +2. Check that the subscription is correct. +3. Refresh the resource picker by re-selecting the subscription. +4. Verify the identity has access to the resource's resource group. + +### Logs query timeout + +**Symptoms:** + +- Query runs for a long time then fails +- Error mentions timeout or query limits + +**Solutions:** + +1. Narrow the time range to reduce data volume. +2. Add filters to reduce the result set. +3. Use `summarize` to aggregate data instead of returning raw rows. +4. Consider using Basic Logs for large datasets (if enabled). +5. Break complex queries into smaller parts. + +### "Metrics not available" for a resource + +**Symptoms:** + +- Resource appears in picker but no metrics are listed +- Metric dropdown is empty + +**Solutions:** + +1. Verify the resource type supports Azure Monitor metrics. +2. Check if the resource is in a region that supports metrics. +3. Some resources require diagnostic settings to emit metrics—configure these in Azure Portal. +4. Try selecting a different namespace for the resource. + +## Azure Resource Graph errors + +These errors are specific to Azure Resource Graph (ARG) queries. + +### "Query execution failed" + +**Symptoms:** + +- ARG query fails with execution errors +- Results don't match expected resources + +**Solutions:** + +1. Validate query syntax in Azure Portal Resource Graph Explorer. +2. Check that you have access to the subscriptions being queried. +3. Verify table names are correct (e.g., `Resources`, `ResourceContainers`). +4. Some ARG features require specific permissions, check [ARG documentation](https://docs.microsoft.com/en-us/azure/governance/resource-graph/). + +### Query returns incomplete results + +**Symptoms:** + +- Not all expected resources appear in results +- Results seem truncated + +**Solutions:** + +1. ARG queries are paginated. The data source handles pagination automatically, but very large result sets may be limited. +2. Add filters to reduce result set size. +3. Verify you have access to all subscriptions containing the resources. + +## Application Insights Traces errors + +These errors are specific to the Traces query type. + +### "No traces found" + +**Symptoms:** + +- Trace query returns empty results +- Operation ID search finds nothing + +**Solutions:** + +1. Verify the Application Insights resource is collecting trace data. +2. Check that the time range includes when the traces were generated. +3. Ensure the Operation ID is correct (copy directly from another trace or log). +4. Verify the identity has access to the Application Insights resource. + +## Template variable errors + +For detailed troubleshooting of template variables, refer to the [template variables troubleshooting section](ref:template-variables). + +### Variables return no values + +**Solutions:** + +1. Verify the data source connection is working (test it in the data source settings). +2. Check that parent variables (for cascading variables) have valid selections. +3. Verify the identity has permissions to list the requested resources. +4. For Logs variables, ensure the KQL query returns a single column. + +### Variables are slow to load + +**Solutions:** + +1. Set variable refresh to **On dashboard load** instead of **On time range change**. +2. Reduce the scope of variable queries (e.g., filter by resource group instead of entire subscription). +3. For Logs variables, optimize the KQL query to return results faster. + +## Connection and network errors + +These errors indicate problems with network connectivity between Grafana and Azure services. + +### "Connection refused" or timeout errors + +**Symptoms:** + +- Data source test fails with network errors +- Queries timeout without returning results + +**Solutions:** + +1. Verify network connectivity from Grafana to Azure endpoints. +2. Check firewall rules allow outbound HTTPS (port 443) to Azure services. +3. For private networks, ensure Private Link or VPN is configured correctly. +4. For Grafana Cloud, configure [Private Data Source Connect](ref:configure-azure-monitor) if accessing private resources. + +### SSL/TLS certificate errors + +**Symptoms:** + +- Certificate validation failures +- SSL handshake errors + +**Solutions:** + +1. Ensure the system time is correct (certificate validation fails with incorrect time). +2. Verify corporate proxy isn't intercepting HTTPS traffic. +3. Check that required CA certificates are installed on the Grafana server. + +## Get additional help + +If you've tried the solutions above and still encounter issues: + +1. Check the [Grafana community forums](https://community.grafana.com/) for similar issues. +1. Review the [Azure Monitor data source GitHub issues](https://github.com/grafana/grafana/issues) for known bugs. +1. Enable debug logging in Grafana to capture detailed error information. +1. Contact Grafana Support if you're an Enterprise, Cloud Pro or Cloud Contracted user. +1. When reporting issues, include: + - Grafana version + - Error messages (redact sensitive information) + - Steps to reproduce + - Relevant configuration (redact credentials)