(cherry picked from commit 327fa124a4)
This commit is contained in:
@@ -11,14 +11,14 @@ An exemplar is a specific trace representative of a repeated pattern of data in
|
||||
|
||||
Suppose your company website is experiencing a surge in traffic volumes. While more than eighty percent of the users are able to access the website in under two seconds, some users are experiencing a higher than normal response time resulting in bad user experience
|
||||
|
||||
To identify the factors that are contributing to the latency, you must compare a trace for a fast response against a trace for a slow response. Given the vast amount of data in a typical production environment, it will be extremely laborious and time-consuming effort.
|
||||
To identify the factors that are contributing to the latency, you must compare a trace for a fast response against a trace for a slow response. Given the vast amount of data in a typical production environment, it will be extremely laborious and time-consuming effort.
|
||||
|
||||
Use exemplars to help isolate problems within your data distribution by pinpointing query traces exhibiting high latency within a time interval. Once you localize the latency problem to a few exemplar traces, you can combine it with additional system based information or location properties to perform a root cause analysis faster, leading to quick resolutions to performance issues.
|
||||
|
||||
Support for exemplars is available for the Prometheus data source only. Once you enable the functionality, exemplars data is available by default. For more information on exemplar configuration and how to enable exemplars, refer to [configuring exemplars in Prometheus data source]({{< relref "../../datasources/prometheus.md#configuring-exemplars" >}}).
|
||||
|
||||
Grafana shows exemplars alongside a metric in the Explore view and in dashboards. Each exemplar displays as a highlighted star. You can hover your cursor over an exemplar to view the unique traceID, which is a combination of a key value pair. To investigate further, click the blue button next to the `traceID` property.
|
||||
Grafana shows exemplars alongside a metric in the Explore view and in dashboards. Each exemplar displays as a highlighted star. You can hover your cursor over an exemplar to view the unique traceID, which is a combination of a key value pair. To investigate further, click the blue button next to the `traceID` property.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/v74/exemplars.png" class="docs-image--no-shadow" max-width= "750px" caption="Screenshot showing the detail window of an Exemplar" >}}
|
||||
{{< figure src="/static/img/docs/v74/exemplars.png" class="docs-image--no-shadow" max-width= "750px" caption="Screenshot showing the detail window of an Exemplar" >}}
|
||||
|
||||
Refer to [View exemplar data]({{< relref "./view-exemplars.md" >}}) for instructions on how to drill down and view exemplar trace details from metrics and logs. To know more about exemplars, refer to the blogpost [Intro to exemplars, which enable Grafana Tempo’s distributed tracing at massive scale](https://grafana.com/blog/2021/03/31/intro-to-exemplars-which-enable-grafana-tempos-distributed-tracing-at-massive-scale/).
|
||||
|
||||
@@ -7,7 +7,7 @@ weight = 400
|
||||
|
||||
# View exemplar data
|
||||
|
||||
When support for exemplar support is enabled for a Prometheus data source, you can view exemplar data either in the Explore view or from the Loki log details.
|
||||
When support for exemplar support is enabled for a Prometheus data source, you can view exemplar data either in the Explore view or from the Loki log details.
|
||||
|
||||
## In Explore
|
||||
|
||||
@@ -15,13 +15,13 @@ Explore visualizes exemplar traces as highlighted stars alongside metrics data.
|
||||
|
||||
To examine the details of an exemplar trace:
|
||||
|
||||
1. Place your cursor over an exemplar (highlighted star). Depending on your backend trace data source, you will see a blue button with the label `Query with <data source name>`. In the following example, the tracing data source is Tempo.
|
||||
1. Place your cursor over an exemplar (highlighted star). Depending on your backend trace data source, you will see a blue button with the label `Query with <data source name>`. In the following example, the tracing data source is Tempo.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/basics/exemplar-details.png" class="docs-image--no-shadow" max-width= "275px" caption="Screenshot showing Exemplar details" >}}
|
||||
{{< figure src="/static/img/docs/basics/exemplar-details.png" class="docs-image--no-shadow" max-width= "275px" caption="Screenshot showing Exemplar details" >}}
|
||||
|
||||
1. Click the **Query with Tempo** option next to the `traceID` property. The trace details, including the spans within the trace are listed in a separate panel on the right.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/basics/exemplar-explore-view.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}}
|
||||
{{< figure src="/static/img/docs/basics/exemplar-explore-view.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}}
|
||||
|
||||
For more information on how to drill down and analyze the trace and span details, refer to the [Analyze trace and span details](#analyze-trace-and-spans) section.
|
||||
|
||||
@@ -32,32 +32,32 @@ You can also view exemplar trace details from the Loki logs in Explore. Use rege
|
||||
|
||||
To view the details of an exemplar trace:
|
||||
|
||||
1. Expand a log line and scroll down to the `Detected fields` section. Depending on your backend trace data source, you will see a blue button with the label `<data source name>`.
|
||||
1. Expand a log line and scroll down to the `Detected fields` section. Depending on your backend trace data source, you will see a blue button with the label `<data source name>`.
|
||||
|
||||
1. Click the blue button next to the `traceID` property. Typically, it will have the name of the backend data source. In the following example, the tracing data source is Tempo. The trace details, including the spans within the trace are listed in a separate panel on the right.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/basics/exemplar-loki-logs.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}}
|
||||
{{< figure src="/static/img/docs/basics/exemplar-loki-logs.png" class="docs-image--no-shadow" max-width= "750px" caption="Explorer view with panel showing trace details" >}}
|
||||
|
||||
For more information on how to drill down and analyze the trace and span details, refer to the [Analyze trace and span details](#analyze-trace-and-spans) section.
|
||||
|
||||
|
||||
## Analyze trace and spans
|
||||
|
||||
This panel shows the details of the trace in different segments.
|
||||
|
||||
- The top segment shows the Trace ID to indicate that the query results correspond to the specific trace.
|
||||
This panel shows the details of the trace in different segments.
|
||||
|
||||
- The top segment shows the Trace ID to indicate that the query results correspond to the specific trace.
|
||||
|
||||
You can add more traces to the results using the `Add query` button.
|
||||
|
||||
|
||||
- The next segment shows the entire span for the specific trace as a narrow strip. All levels of the trace from the client all the way down to database query is displayed, which provides a bird's eye view of the time distribution across all layers over which the HTTP request was processed.
|
||||
|
||||
1. You can click within this strip view to display a magnified view of a smaller time segment within the span. This magnified view shows up in the bottom segment of the panel.
|
||||
|
||||
1. In the magnified view, you can expand or collapse the various levels of the trace to drill down to the specific span of interest.
|
||||
1. In the magnified view, you can expand or collapse the various levels of the trace to drill down to the specific span of interest.
|
||||
|
||||
For example, if the strip view shows that most of the latency was within the app layer, you can expand the trace down the app layer to investigate the problem further. To expand a particular layer of span, click the icon on the left. The same button can collapse an expanded span.
|
||||
|
||||
- To see the details of the span at any level, click the span itself.
|
||||
- To see the details of the span at any level, click the span itself.
|
||||
|
||||
This displays additional metadata associated with the span. The metadata itself is initially shown in a narrow strip but you can see more details by clicking the metadata strip.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/basics/exemplar-span-details.png" class="docs-image--no-shadow" max-width= "750px" caption="Span details" >}}
|
||||
{{< figure src="/static/img/docs/basics/exemplar-span-details.png" class="docs-image--no-shadow" max-width= "750px" caption="Span details" >}}
|
||||
|
||||
@@ -19,7 +19,7 @@ and the bar height represents the frequency (such as count) of values that fell
|
||||
This histogram shows the value distribution of a couple of time series. You can easily see that
|
||||
most values land between 240-300 with a peak between 260-280.
|
||||
|
||||

|
||||

|
||||
|
||||
Histograms only look at _value distributions_ over a specific time range. The problem with histograms is you cannot see any trends or changes in the distribution over time.
|
||||
This is where heatmaps become useful.
|
||||
@@ -30,7 +30,7 @@ A _heatmap_ is like a histogram, but over time where each time slice represents
|
||||
|
||||
In this example, you can clearly see what values are more common and how they trend over time.
|
||||
|
||||

|
||||

|
||||
|
||||
## Pre-bucketed data
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ In [Introduction to time series]({{< relref "timeseries.md#time-series-databases
|
||||
|
||||
With time series data, the data often contain more than a single series, and is a set of multiple time series. Many Grafana data sources support this type of data.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/example_graph_multi_dim.png" class="docs-image--no-shadow" max-width="850px" >}}
|
||||
{{< figure src="/static/img/docs/example_graph_multi_dim.png" class="docs-image--no-shadow" max-width="850px" >}}
|
||||
|
||||
The common case is issuing a single query for a measurement with one or more additional properties as dimensions. For example, querying a temperature measurement along with a location property. In this case, multiple series are returned back from that single query and each series has unique location as a dimension.
|
||||
|
||||
|
||||
@@ -19,7 +19,7 @@ Temperature data like this is one example of what we call a *time series*—a se
|
||||
|
||||
Tables are useful when you want to identify individual measurements but make it difficult to see the big picture. A more common visualization for time series is the _graph_, which instead places each measurement along a time axis. Visual representations like the graph make it easier to discover patterns and features of the data that otherwise would be difficult to see.
|
||||
|
||||
{{< docs-imagebox img="/img/docs/example_graph.png" class="docs-image--no-shadow" max-width="850px" >}}
|
||||
{{< figure src="/static/img/docs/example_graph.png" class="docs-image--no-shadow" max-width="850px" >}}
|
||||
|
||||
Temperature data like the one in the example, is far from the only example of a time series. Other examples of time series are:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user