From d1f1beaef9b05ab2fd8f0fdccce4bc7acbd8f870 Mon Sep 17 00:00:00 2001 From: Giuseppe Guerra Date: Fri, 14 Apr 2023 12:18:07 +0200 Subject: [PATCH] Docs: Add distributed tracing plugin developer guide (#65898) * Docs: Add plugin developer guide on distributed tracing * Docs: Fix typo * Docs: Plugins: Tracing: Address TODOs and other changes * Docs: Plugins: Tracing: Fix typo * Docs: Plugins: Tracing: Add note about min SDK version and update command * Docs: Plugins: Tracing: Fix some typos, rephrase some sentences * Docs: Plugins: Traacing: Removed manual instance management instructions * Update docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md Co-authored-by: Joseph Perez <45749060+josmperez@users.noreply.github.com> * Docs: Plugins: Tracing: Use markdown instead of HTML --------- Co-authored-by: Joseph Perez <45749060+josmperez@users.noreply.github.com> --- docs/sources/developers/plugins/_index.md | 1 + ...distributed-tracing-for-backend-plugins.md | 123 ++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md diff --git a/docs/sources/developers/plugins/_index.md b/docs/sources/developers/plugins/_index.md index 31739b21e3d..8011411857b 100644 --- a/docs/sources/developers/plugins/_index.md +++ b/docs/sources/developers/plugins/_index.md @@ -52,6 +52,7 @@ Improve an existing plugin with one of our guides: - [Add support for Explore queries]({{< relref "add-support-for-explore-queries.md" >}}) - [Add support for variables]({{< relref "add-support-for-variables.md" >}}) - [Add a query editor help component]({{< relref "add-query-editor-help.md" >}}) +- [Add distributed tracing for backend plugins]({{< relref "add-distributed-tracing-for-backend-plugins.md" >}}) - [Build a logs data source plugin]({{< relref "build-a-logs-data-source-plugin.md" >}}) - [Build a streaming data source plugin]({{< relref "build-a-streaming-data-source-plugin.md" >}}/) - [Error handling]({{< relref "error-handling.md" >}}) diff --git a/docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md b/docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md new file mode 100644 index 00000000000..17ea19108ea --- /dev/null +++ b/docs/sources/developers/plugins/add-distributed-tracing-for-backend-plugins.md @@ -0,0 +1,123 @@ +--- +title: Add distributed tracing for backend plugins +--- + +# Add distributed tracing for backend plugins + +> **Note:** This feature requires at least Grafana 9.5.0, and your plugin needs to be built at least with +> grafana-plugins-sdk-go v0.157.0. If you run a plugin with tracing features on an older version of Grafana, +> tracing will be disabled. + +## Introduction + +Distributed tracing allows backend plugin developers to create custom spans in their plugins, and send them to the same endpoint +and with the same propagation format as the main Grafana instance. The tracing context is also propagated from the Grafana instance +to the plugin, so the plugin's spans will be correlated to the correct trace. + +## Configuration + +> **Note:** Only OpenTelemetry is supported. If Grafana is configured to use a deprecated tracing system (Jaeger or OpenTracing), +> tracing will be disabled in the plugin. Please note that OpenTelemetry + Jaeger propagator is supported. + +OpenTelemetry must be enabled and configured for the Grafana instance. Please refer to [this section]( +{{< relref "../../setup-grafana/configure-grafana/#tracingopentelemetry" >}}) for more information. + +As of Grafana 9.5.0, plugins tracing must be enabled manually on a per-plugin basis, by specifying `tracing = true` in the plugin's config section: + +```ini +[plugin.myorg-myplugin-datasource] +tracing = true +``` + +## Implementing tracing in your plugin + +> **Note:** Make sure you are using at least grafana-plugin-sdk-go v0.157.0. You can update with `go get -u github.com/grafana/grafana-plugin-sdk-go`. + +When OpenTelemetry tracing is enabled on the main Grafana instance and tracing is enabeld for a plugin, +the OpenTelemetry endpoint address and propagation format will be passed to the plugin during startup, +which will be used to configure a global tracer. + +1. The global tracer is configured automatically if you use datasource.Manage or app.Manage to run your plugin. + + This also allows you to specify custom attributes for the default tracer: + + ```go + func main() { + if err := datasource.Manage("MY_PLUGIN_ID", plugin.NewDatasource, datasource.ManageOpts{ + TracingOpts: tracing.Opts{ + // Optional custom attributes attached to the tracer's resource. + // The tracer will already have some SDK and runtime ones pre-populated. + CustomAttributes: []attribute.KeyValue{ + attribute.String("my_plugin.my_attribute", "custom value"), + }, + }, + }); err != nil { + log.DefaultLogger.Error(err.Error()) + os.Exit(1) + } + } + ``` + +1. Once tracing is configured, you can access the global tracer with: + + ```go + tracing.DefaultTracer() + ``` + + this returns an [OpenTelemetry trace.Tracer](https://pkg.go.dev/go.opentelemetry.io/otel/trace#Tracer), and can be used to create spans. + + For example: + + ```go + func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) { + ctx, span := tracing.DefaultTracer().Start( + ctx, + "query processing", + trace.WithAttributes( + attribute.String("query.ref_id", query.RefID), + attribute.String("query.type", query.QueryType), + attribute.Int64("query.max_data_points", query.MaxDataPoints), + attribute.Int64("query.interval_ms", query.Interval.Milliseconds()), + attribute.Int64("query.time_range.from", query.TimeRange.From.Unix()), + attribute.Int64("query.time_range.to", query.TimeRange.To.Unix()), + ), + ) + defer span.End() + log.DefaultLogger.Debug("query", "traceID", trace.SpanContextFromContext(ctx).TraceID()) + + // ... + } + ``` + + Refer to the [OpenTelemetry Go SDK](https://pkg.go.dev/go.opentelemetry.io/otel) for in-depth documentation about all the features provided by OpenTelemetry. + + If tracing is disabled in Grafana, `backend.DefaultTracer()` returns a no-op tracer. + +### Tracing GRPC calls + +A new span is created automatically for each GRPC call (`QueryData`, `CheckHealth`, etc), both on Grafana's side and +on the plugin's side. + +This also injects the trace context into the `context.Context` passed to those methods. + +This allows you to retrieve the [trace.SpanContext](https://pkg.go.dev/go.opentelemetry.io/otel/trace#SpanContext) by using `tracing.SpanContextFromContext` by passing the original `context.Context` to it: + +```go +func (d *Datasource) query(ctx context.Context, pCtx backend.PluginContext, query backend.DataQuery) (backend.DataResponse, error) { + spanCtx := trace.SpanContextFromContext(ctx) + traceID := spanCtx.TraceID() + + // ... +} +``` + +### Tracing HTTP requests + +When tracing is enabled, a `TracingMiddleware` is also added to the default middleware stack to all HTTP clients created +using the `httpclient.New` or `httpclient.NewProvider`, unless custom middlewares are specified. + +This middleware creates spans for each outgoing HTTP request and provides some useful attributes and events related to the request's lifecycle. + +## Complete plugin example + +You can refer to the [datasource-http-backend plugin example](https://github.com/grafana/grafana-plugin-examples/tree/main/examples/datasource-http-backend) for a complete example of a plugin that has full tracing support.