From 3f78facce0d4743ca178e065937c4ea8dfbf106f Mon Sep 17 00:00:00 2001 From: Larissa Wandzura Date: Tue, 6 Jan 2026 15:55:56 -0600 Subject: [PATCH] updated the configure doc --- docs/sources/datasources/loki/_index.md | 6 +- .../configure/configure-loki-data-source.md | 215 ----------- .../datasources/loki/configure/index.md | 355 ++++++++++++++++++ 3 files changed, 358 insertions(+), 218 deletions(-) delete mode 100644 docs/sources/datasources/loki/configure/configure-loki-data-source.md create mode 100644 docs/sources/datasources/loki/configure/index.md diff --git a/docs/sources/datasources/loki/_index.md b/docs/sources/datasources/loki/_index.md index 10f2a13a297..1224fe8555c 100644 --- a/docs/sources/datasources/loki/_index.md +++ b/docs/sources/datasources/loki/_index.md @@ -99,7 +99,7 @@ The Loki data source provides the following capabilities: The following documentation helps you get started with the Loki data source: -- [Configure the Loki data source](configure/configure-loki-data-source/) +- [Configure the Loki data source](configure/) - [Loki query editor](query-editor/) - [Loki template variables](template-variables/) - [Troubleshoot the Loki data source](troubleshooting/) @@ -121,7 +121,7 @@ After you configure the Loki data source, you can: - Add [annotations](ref:annotate-visualizations) to overlay log events on graphs - Set up [alerting](ref:alerting) to monitor your log data - Use [Explore](ref:explore) for ad-hoc log queries and analysis -- Configure [derived fields](configure/configure-loki-data-source/#derived-fields) to link logs to traces or other data sources +- Configure [derived fields](configure/#derived-fields) to link logs to traces or other data sources ## Community dashboards @@ -140,7 +140,7 @@ For more information, refer to [Import a dashboard](ref:import-dashboard). Loki integrates with other Grafana data sources to provide full observability across logs, metrics, and traces: -- **Tempo:** Use [derived fields](configure/configure-loki-data-source/#derived-fields) to create links from log lines to traces in Tempo, enabling seamless navigation from logs to distributed traces. +- **Tempo:** Use [derived fields](configure/#derived-fields) to create links from log lines to traces in Tempo, enabling seamless navigation from logs to distributed traces. - **Prometheus and Mimir:** Display logs alongside metrics on the same dashboard to correlate application behavior with performance data. For more information about building observability workflows, refer to the [Grafana Tempo documentation](https://grafana.com/docs/tempo/latest/) and [Grafana Mimir documentation](https://grafana.com/docs/mimir/latest/). diff --git a/docs/sources/datasources/loki/configure/configure-loki-data-source.md b/docs/sources/datasources/loki/configure/configure-loki-data-source.md deleted file mode 100644 index 5fe6e644089..00000000000 --- a/docs/sources/datasources/loki/configure/configure-loki-data-source.md +++ /dev/null @@ -1,215 +0,0 @@ ---- -aliases: - - ../data-sources/loki/ - - ../features/datasources/loki/ -description: Configure the Loki data source -keywords: - - grafana - - loki - - logging - - guide - - data source -menuTitle: Configure Loki -title: Configure the Loki data source -weight: 200 -refs: - log-details: - - pattern: /docs/grafana/ - destination: /docs/grafana//explore/logs-integration/#labels-and-detected-fields - - pattern: /docs/grafana-cloud/ - destination: /docs/grafana//explore/logs-integration/#labels-and-detected-fields ---- - -# Loki data source - -Grafana ships with built-in support for [Loki](/docs/loki/latest/), an open-source log aggregation system by Grafana Labs. If you are new to Loki the following documentation will help you get started: - -- [Getting started](/docs/loki/latest/get-started/) -- [Best practices](/docs/loki/latest/best-practices/#best-practices) - -## Configure the Loki data source - -To add the Loki data source, complete the following steps: - -1. Click **Connections** in the left-side menu. -1. Under **Connections**, click **Add new connection**. -1. Enter `Loki` in the search bar. -1. Select **Loki data source**. -1. Click **Create a Loki data source** in the upper right. - -You will be taken to the **Settings** tab where you will set up your Loki configuration. - -## Configuration options - -The following is a list of configuration options for Loki. - -The first option to configure is the name of your connection: - -- **Name** - The data source name. This is how you refer to the data source in panels and queries. Examples: loki-1, loki_logs. - -- **Default** - Toggle to select as the default name in dashboard panels. When you go to a dashboard panel this will be the default selected data source. - -### HTTP section - -- **URL** - The URL of your Loki server. Loki uses port 3100. If your Loki server is local, use `http://localhost:3100`. If it is on a server within a network, this is the URL with port where you are running Loki. Example: `http://loki.example.orgname:3100`. - -- **Allowed cookies** - Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default. - -- **Timeout** - The HTTP request timeout. This must be in seconds. There is no default, so this setting is up to you. - -### Auth section - -There are several authentication methods you can choose in the Authentication section. - -{{< admonition type="note" >}} -Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For information on setting up TLS encryption with Loki see [Grafana Loki configuration parameters](/docs/loki/latest/configuration/). -{{< /admonition >}} - -- **Basic authentication** - The most common authentication method. Use your `data source` user name and `data source` password to connect. - -- **With credentials** - Toggle on to enable credentials such as cookies or auth headers to be sent with cross-site requests. - -- **TLS client authentication** - Toggle on to use client authentication. When enabled, add the `Server name`, `Client cert` and `Client key`. The client provides a certificate that is validated by the server to establish the client's trusted identity. The client key encrypts the data between client and server. - -- **With CA cert** - Authenticate with a CA certificate. Follow the instructions of the CA (Certificate Authority) to download the certificate file. - -- **Skip TLS verify** - Toggle on to bypass TLS certificate validation. - -- **Forward OAuth identity** - Forward the OAuth access token (and also the OIDC ID token if available) of the user querying the data source. - -### Custom HTTP headers - -- **Header** - Add a custom header. This allows custom headers to be passed based on the needs of your Loki instance. - -- **Value** - The value of the header. - -### Alerting - -- **Manage alert rules in Alerting UI** - Toggle on to manage alert rules for the Loki data source. To manage other alerting resources add an `Alertmanager` data source. - -### Queries - -- **Maximum lines** - Sets the maximum number of log lines returned by Loki. Increase the limit to have a bigger results set for ad-hoc analysis. Decrease the limit if your browser is sluggish when displaying log results. The default is `1000`. - - - -### Derived fields - -Derived Fields are used to extract new fields from your logs and create a link from the value of the field. - -For example, you can link to your tracing backend directly from your logs, or link to a user profile page if the log line contains a corresponding `userId`. -These links appear in the [log details](ref:log-details). - -You can add multiple derived fields. - -{{< admonition type="note" >}} -If you use Grafana Cloud, you can request modifications to this feature by clicking **Open a Support Ticket** from the Grafana Cloud Portal. -{{< /admonition >}} - -Each derived field consists of the following: - -- **Name** - Sets the field name. Displayed as a label in the log details. - -- **Type** - Defines the type of the derived field. It can be either: - -{{< admonition type="caution" >}} -Using complex regular expressions in either type can impact browser performance when processing large volumes of logs. Consider using simpler patterns when possible. -{{< /admonition >}} - -- **Regex**: A regular expression to parse a part of the log message and capture it as the value of the new field. Can contain only one capture group. - -- **Label**: A label from the selected log line. This can be any type of label - indexed, parsed or structured metadata. When using this type, the input will match as a regular expression against label keys, allowing you to match variations like `traceid` and `trace_id` with a single regex pattern like `trace[_]?id`. The value of the matched label will be used as the value of the derived field. - -- **URL/query** Sets the full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro. - -- **URL Label** - Sets a custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source. - -- **Internal link** - Toggle on to define an internal link. For internal links, you can select the target data source from a selector. This supports only tracing data sources. - -- **Open in new tab** - Toggle on to open the link in a new tab or window. - -- **Show example log message** - Click to paste an example log line to test the regular expression of your derived fields. - -Click **Save & test** to test your connection. - -#### Troubleshoot interpolation - -You can use a debug section to see what your fields extract and how the URL is interpolated. -Select **Show example log message** to display a text area where you can enter a log message. - -{{< figure src="/static/img/docs/v75/loki_derived_fields_settings.png" class="docs-image--no-shadow" max-width="800px" caption="Screenshot of the derived fields debugging" >}} - -The new field with the link shown in log details: - -{{< figure src="/static/img/docs/explore/data-link-9-4.png" max-width="800px" caption="Data link in Explore" >}} - -## Provision the data source - -You can define and configure the data source in YAML files as part of Grafana's provisioning system. -For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources). - -### Provisioning examples - -```yaml -apiVersion: 1 - -datasources: - - name: Loki - type: loki - access: proxy - url: http://localhost:3100 - jsonData: - timeout: 60 - maxLines: 1000 -``` - -**Using basic authorization and a derived field:** - -You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables: - -```yaml -apiVersion: 1 - -datasources: - - name: Loki - type: loki - access: proxy - url: http://localhost:3100 - basicAuth: true - basicAuthUser: my_user - jsonData: - maxLines: 1000 - derivedFields: - # Field with internal link pointing to data source in Grafana. - # datasourceUid value can be anything, but it should be unique across all defined data source uids. - - datasourceUid: my_jaeger_uid - matcherRegex: "traceID=(\\w+)" - name: TraceID - # url will be interpreted as query for the datasource - url: '$${__value.raw}' - # optional for URL Label to set a custom display label for the link. - urlDisplayLabel: 'View Trace' - - # Field with external link. - - matcherRegex: "traceID=(\\w+)" - name: TraceID - url: 'http://localhost:16686/trace/$${__value.raw}' - secureJsonData: - basicAuthPassword: test_password -``` - -**Using a Jaeger data source:** - -In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value. - -``` -datasources: - - name: Jaeger - type: jaeger - url: http://jaeger-tracing-query:16686/ - access: proxy - # UID should match the datasourceUid in derivedFields. - uid: my_jaeger_uid -``` \ No newline at end of file diff --git a/docs/sources/datasources/loki/configure/index.md b/docs/sources/datasources/loki/configure/index.md new file mode 100644 index 00000000000..707ea561830 --- /dev/null +++ b/docs/sources/datasources/loki/configure/index.md @@ -0,0 +1,355 @@ +--- +aliases: + - ../data-sources/loki/ + - ../features/datasources/loki/ + - ../configure-loki-data-source/ +description: Configure the Loki data source +keywords: + - grafana + - loki + - logging + - guide + - data source +menuTitle: Configure +title: Configure the Loki data source +weight: 200 +refs: + log-details: + - pattern: /docs/grafana/ + destination: /docs/grafana//explore/logs-integration/#labels-and-detected-fields + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//explore/logs-integration/#labels-and-detected-fields + alerting: + - pattern: /docs/grafana/ + destination: /docs/grafana//alerting/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/alerting-and-irm/alerting/ + private-data-source-connect: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/ + configure-pdc: + - pattern: /docs/grafana/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc + provisioning-data-sources: + - pattern: /docs/grafana/ + destination: /docs/grafana//administration/provisioning/#data-sources + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//administration/provisioning/#data-sources + data-source-management: + - pattern: /docs/grafana/ + destination: /docs/grafana//administration/data-source-management/ + - pattern: /docs/grafana-cloud/ + destination: /docs/grafana//administration/data-source-management/ +--- + +# Configure the Loki data source + +This document provides instructions for configuring the Loki data source and explains available configuration options. For general information about data sources, refer to [Data source management](ref:data-source-management). + +Grafana ships with built-in support for [Loki](https://grafana.com/docs/loki/latest/), an open-source log aggregation system by Grafana Labs. If you are new to Loki, the following documentation will help you get started: + +- [Getting started](https://grafana.com/docs/loki/latest/get-started/) +- [Best practices](https://grafana.com/docs/loki/latest/best-practices/#best-practices) + +## Before you begin + +Before configuring the Loki data source, ensure you have the following: + +- **Grafana permissions:** You must have the `Organization administrator` role to configure data sources. Organization administrators can also [configure the data source via YAML](#provision-the-data-source) with the Grafana provisioning system or [using Terraform](#provision-the-data-source-using-terraform). + +- **Loki instance:** You need a running Loki instance and its URL. If you don't have one, refer to the [Loki installation documentation](https://grafana.com/docs/loki/latest/setup/install/). + +- **Authentication details (if applicable):** If your Loki instance requires authentication, gather the necessary credentials such as username and password for basic authentication, or any required certificates for TLS authentication. + +{{< admonition type="note" >}} +The Loki data source plugin is built into Grafana. No additional installation is required. +{{< /admonition >}} + +## Add the Loki data source + +To add the Loki data source, complete the following steps: + +1. Click **Connections** in the left-side menu. +1. Under **Connections**, click **Add new connection**. +1. Enter `Loki` in the search bar. +1. Select **Loki data source**. +1. Click **Create a Loki data source** in the upper right. + +You are taken to the **Settings** tab where you will set up your Loki configuration. + +## Configure Loki using the UI + +The following are the configuration options for Loki. + +| Name | Description | +| ---- | ----------- | +| **Name** | The data source name. This is how you refer to the data source in panels and queries. Examples: `loki-1`, `loki_logs`. | +| **Default** | Toggle to set this data source as the default. When enabled, new panels automatically use this data source. | + +### Connection section + +| Name | Description | +| ---- | ----------- | +| **URL** | The URL of your Loki server, including the port. The default Loki port is `3100`. Examples: `http://localhost:3100`, `http://loki.example.org:3100`. | + +### Authentication section + +Select an authentication method from the **Authentication** dropdown. + +| Setting | Description | +| ---- | ----------- | +| **No authentication** | No authentication is required to access the data source. | +| **Basic authentication** | Authenticate using a username and password. Enter the credentials in the **User** and **Password** fields. | +| **Forward OAuth identity** | Forward the OAuth access token (and the OIDC ID token if available) of the user querying the data source. | + +### TLS settings + +Use TLS (Transport Layer Security) for an additional layer of security when working with Loki. For more information on setting up TLS encryption with Loki, refer to [Grafana Loki configuration parameters](https://grafana.com/docs/loki/latest/configuration/). + +| Setting | Description | +| ---- | ----------- | +| **Add self-signed certificate** | Enable to add a self-signed CA certificate. When enabled, enter the certificate in the **CA Certificate** field. The certificate must begin with `-----BEGIN CERTIFICATE-----`. | +| **TLS Client Authentication** | Enable to use client certificate authentication. When enabled, enter the **ServerName** (for example, `domain.example.com`), **Client Certificate** (begins with `-----BEGIN CERTIFICATE-----`), and **Client Key** (begins with `-----BEGIN RSA PRIVATE KEY-----`). | +| **Skip TLS certificate validation** | Enable to bypass TLS certificate validation. Use this option only for testing or when connecting to Loki instances with self-signed certificates. | + +### HTTP headers + +Use HTTP headers to pass along additional context and metadata about the request/response. + +| Setting | Description | +| ---- | ----------- | +| **Header** | The name of the custom header. For example, `X-Custom-Header`. | +| **Value** | The value of the custom header. For example, `Header value`. | + +Click **+ Add another header** to add additional headers. + +## Additional settings + +Additional settings are optional settings that you can configure for more control over your data source. + +### Advanced HTTP settings + +| Setting | Description | +| ------- | ----------- | +| **Allowed cookies** | Specify cookies by name that should be forwarded to the data source. The Grafana proxy deletes all forwarded cookies by default. | +| **Timeout** | The HTTP request timeout in seconds. If not set, the default Grafana timeout is used. | + +### Alerting + +Manage alert rules for the Loki data source. For more information, refer to [Alerting](ref:alerting). + +| Setting | Description | +| ------- | ----------- | +| **Manage alert rules in Alerting UI** | Toggle to manage alert rules for this Loki data source in the Grafana Alerting UI. | + +### Queries + +Configure options to customize your querying experience. + +| Setting | Description | +| ------- | ----------- | +| **Maximum lines** | The maximum number of log lines returned by Loki. The default is `1000`. Increase for larger result sets during ad-hoc analysis. Decrease if your browser is sluggish when displaying log results. | + +### Derived fields + +Derived fields can be used to extract new fields from a log message and create a link from its value. For example, you can link to your tracing backend directly from your logs. These links appear in the [log details](ref:log-details). + +Click **+ Add** to add a derived field. Each derived field has the following settings: + +| Setting | Description | +| ------- | ----------- | +| **Name** | The field name. Displayed as a label in the log details. | +| **Type** | The type of derived field. Select **Regex in log line** to extract values using a regular expression, or **Label** to use an existing label value. | +| **Regex** | A regular expression to parse a part of the log message and capture it as the value of the new field. Can contain only one capture group. | +| **URL** | The full link URL if the link is external, or a query for the target data source if the link is internal. You can interpolate the value from the field with the `${__value.raw}` macro. For example, `http://example.com/${__value.raw}`. | +| **URL Label** | A custom display label for the link. This setting overrides the link label, which defaults to the full external URL or name of the linked internal data source. | +| **Internal link** | Toggle to define an internal link. When enabled, you can select the target data source from a selector. This supports only tracing data sources. | +| **Open in new tab** | Toggle to open the link in a new browser tab or window. | + +{{< admonition type="caution" >}} +Using complex regular expressions can impact browser performance when processing large volumes of logs. Consider using simpler patterns when possible. +{{< /admonition >}} + +#### Test derived fields + +To test your derived field configuration: + +1. Click **Show example log message** to display the debug section. +1. In the **Debug log message** field, paste an example log line to test the regular expressions of your derived fields. +1. Verify that the field extracts the expected value and the URL is interpolated correctly. + +### Private data source connect + +_Only for Grafana Cloud users._ + +Private data source connect, or PDC, allows you to establish a private, secured connection between a Grafana Cloud instance, or stack, and data sources secured within a private network. Click the drop-down to locate the URL for PDC. For more information regarding Grafana PDC, refer to [Private data source connect (PDC)](ref:private-data-source-connect) and [Configure Grafana private data source connect (PDC)](ref:configure-pdc) for instructions on setting up a PDC connection. + +Click **Manage private data source connect** to open your PDC connection page and view your configuration details. + +## Verify the connection + +After configuring the data source, click **Save & test** to save your settings and verify the connection. A successful connection displays the following message: + +**Data source successfully connected.** + +If the test fails, verify: + +- The Loki URL is correct and accessible from the Grafana server +- Any required authentication credentials are correct +- Network connectivity and firewall rules allow the connection +- TLS certificates are valid (if using HTTPS) + +## Provision the data source + +You can define and configure the data source in YAML files as part of the Grafana provisioning system. +For more information about provisioning, and for available configuration options, refer to [Provisioning Grafana](ref:provisioning-data-sources). + +### Provisioning examples + +```yaml +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://localhost:3100 + jsonData: + timeout: 60 + maxLines: 1000 +``` + +**Using basic authorization and a derived field:** + +You must escape the dollar (`$`) character in YAML values because it can be used to interpolate environment variables: + +```yaml +apiVersion: 1 + +datasources: + - name: Loki + type: loki + access: proxy + url: http://localhost:3100 + basicAuth: true + basicAuthUser: my_user + jsonData: + maxLines: 1000 + derivedFields: + # Field with internal link pointing to data source in Grafana. + # datasourceUid value can be anything, but it should be unique across all defined data source uids. + - datasourceUid: my_jaeger_uid + matcherRegex: "traceID=(\\w+)" + name: TraceID + # url will be interpreted as query for the datasource + url: '$${__value.raw}' + # optional for URL Label to set a custom display label for the link. + urlDisplayLabel: 'View Trace' + + # Field with external link. + - matcherRegex: "traceID=(\\w+)" + name: TraceID + url: 'http://localhost:16686/trace/$${__value.raw}' + secureJsonData: + basicAuthPassword: test_password +``` + +**Using a Jaeger data source:** + +In this example, the Jaeger data source's `uid` value should match the Loki data source's `datasourceUid` value. + +```yaml +apiVersion: 1 + +datasources: + - name: Jaeger + type: jaeger + url: http://jaeger-tracing-query:16686/ + access: proxy + # UID should match the datasourceUid in derivedFields. + uid: my_jaeger_uid +``` + +## Provision the data source using Terraform + +You can provision the Loki data source using [Terraform](https://www.terraform.io/) with the [Grafana Terraform provider](https://registry.terraform.io/providers/grafana/grafana/latest/docs). + +For more information about provisioning resources with Terraform, refer to the [Grafana as code using Terraform](https://grafana.com/docs/grafana-cloud/developer-resources/infrastructure-as-code/terraform/) documentation. + +### Basic Terraform example + +The following example creates a basic Loki data source: + +```hcl +resource "grafana_data_source" "loki" { + name = "Loki" + type = "loki" + url = "http://localhost:3100" + + json_data_encoded = jsonencode({ + maxLines = 1000 + }) +} +``` + +### Terraform example with derived fields + +The following example creates a Loki data source with a derived field that links to a Jaeger data source for trace correlation: + +```hcl +resource "grafana_data_source" "loki_with_tracing" { + name = "Loki" + type = "loki" + url = "http://localhost:3100" + + json_data_encoded = jsonencode({ + maxLines = 1000 + derivedFields = [ + { + datasourceUid = grafana_data_source.jaeger.uid + matcherRegex = "traceID=(\\w+)" + name = "TraceID" + url = "$${__value.raw}" + urlDisplayLabel = "View Trace" + } + ] + }) +} +``` + +### Terraform example with basic authentication + +The following example includes basic authentication: + +```hcl +resource "grafana_data_source" "loki_auth" { + name = "Loki" + type = "loki" + url = "http://localhost:3100" + + basic_auth_enabled = true + basic_auth_username = "loki_user" + + secure_json_data_encoded = jsonencode({ + basicAuthPassword = var.loki_password + }) + + json_data_encoded = jsonencode({ + maxLines = 1000 + }) +} +``` + +For all available configuration options, refer to the [Grafana provider data source resource documentation](https://registry.terraform.io/providers/grafana/grafana/latest/docs/resources/data_source). + +## Next steps + +After configuring your Loki data source, explore these resources: + +- [Query the Loki data source](../query-editor/) to learn how to build LogQL queries in Grafana +- [Use template variables](../template-variables/) to create dynamic, reusable dashboards +- [LogQL documentation](https://grafana.com/docs/loki/latest/query/) to learn more about the Loki query language \ No newline at end of file