diff --git a/docs/sources/features/datasources/stackdriver.md b/docs/sources/features/datasources/stackdriver.md index 103a58587b1..3ae2ed3df40 100644 --- a/docs/sources/features/datasources/stackdriver.md +++ b/docs/sources/features/datasources/stackdriver.md @@ -35,7 +35,9 @@ Grafana ships with built-in support for Google Stackdriver. Just add it as a dat ## Authentication -### Service Account Credentials - Private Key File +There are two ways to authenticate the Stackdriver plugin - either by uploading a Google JWT file, or by automatically retrieving credentials from Google metadata server. The latter option is only available when running Grafana on GCE virtual machine. + +### Using a Google Service Account Key File To authenticate with the Stackdriver API, you need to create a Google Cloud Platform (GCP) Service Account for the Project you want to show data for. A Grafana datasource integrates with one GCP Project. If you want to visualize data from multiple GCP Projects then you need to create one datasource per GCP Project. @@ -74,6 +76,16 @@ Click on the links above and click the `Enable` button: {{< docs-imagebox img="/img/docs/v53/stackdriver_grafana_key_uploaded.png" class="docs-image--no-shadow" caption="Service key file is uploaded to Grafana" >}} +### Using GCE Default Service Account + +If Grafana is running on a Google Compute Engine (GCE) virtual machine, it is possible for Grafana to automatically retrieve default credentials from the metadata server. This has the advantage of not needing to generate a private key file for the service account and also not having to upload the file to Grafana. However for this to work, there are a few preconditions that need to be met. + +1. First of all, you need to create a Service Account that can be used by the GCE virtual machine. See detailed instructions on how to do that [here](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#createanewserviceaccount). +2. Make sure the GCE virtual machine instance is being run as the service account that you just created. See instructions [here](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances#using). +3. Allow access to the `Stackdriver Monitoring API` scope. See instructions [here](changeserviceaccountandscopes). + +Read more about creating and enabling service accounts for GCE VM instances [here](https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances). + ## Metric Query Editor {{< docs-imagebox img="/img/docs/v53/stackdriver_query_editor.png" max-width= "400px" class="docs-image--right" >}} @@ -194,7 +206,7 @@ Example Result: `monitoring.googleapis.com/uptime_check/http_status has this val It's now possible to configure datasources using config files with Grafana's provisioning system. You can read more about how it works and all the settings you can set for datasources on the [provisioning docs page](/administration/provisioning/#datasources) -Here is a provisioning example for this datasource. +Here is a provisioning example using the JWT (Service Account key file) authentication type. ```yaml apiVersion: 1 @@ -206,6 +218,7 @@ datasources: jsonData: tokenUri: https://oauth2.googleapis.com/token clientEmail: stackdriver@myproject.iam.gserviceaccount.com + authenticationType: jwt defaultProject: my-project-name secureJsonData: privateKey: | @@ -215,3 +228,16 @@ datasources: yA+23427282348234= -----END PRIVATE KEY----- ``` + +Here is a provisioning example using GCE Default Service Account authentication. + +```yaml +apiVersion: 1 + +datasources: + - name: Stackdriver + type: stackdriver + access: proxy + jsonData: + authenticationType: gce +``` diff --git a/pkg/api/pluginproxy/ds_auth_provider.go b/pkg/api/pluginproxy/ds_auth_provider.go index c68da839d13..edf43085c7c 100644 --- a/pkg/api/pluginproxy/ds_auth_provider.go +++ b/pkg/api/pluginproxy/ds_auth_provider.go @@ -12,6 +12,7 @@ import ( m "github.com/grafana/grafana/pkg/models" "github.com/grafana/grafana/pkg/plugins" "github.com/grafana/grafana/pkg/util" + "golang.org/x/oauth2/google" ) //ApplyRoute should use the plugin route data to set auth headers and custom headers @@ -54,15 +55,30 @@ func ApplyRoute(ctx context.Context, req *http.Request, proxyPath string, route } } - if route.JwtTokenAuth != nil { + authenticationType := ds.JsonData.Get("authenticationType").MustString("jwt") + if route.JwtTokenAuth != nil && authenticationType == "jwt" { if token, err := tokenProvider.getJwtAccessToken(ctx, data); err != nil { logger.Error("Failed to get access token", "error", err) } else { req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", token)) } } - logger.Info("Requesting", "url", req.URL.String()) + if authenticationType == "gce" { + tokenSrc, err := google.DefaultTokenSource(ctx, route.JwtTokenAuth.Scopes...) + if err != nil { + logger.Error("Failed to get default token from meta data server", "error", err) + } else { + token, err := tokenSrc.Token() + if err != nil { + logger.Error("Failed to get default access token from meta data server", "error", err) + } else { + req.Header.Add("Authorization", fmt.Sprintf("Bearer %s", token.AccessToken)) + } + } + } + + logger.Info("Requesting", "url", req.URL.String()) } func interpolateString(text string, data templateData) (string, error) { diff --git a/pkg/tsdb/stackdriver/ensure_default_project.go b/pkg/tsdb/stackdriver/ensure_default_project.go new file mode 100644 index 00000000000..265fdda5151 --- /dev/null +++ b/pkg/tsdb/stackdriver/ensure_default_project.go @@ -0,0 +1,24 @@ +package stackdriver + +import ( + "context" + + "github.com/grafana/grafana/pkg/components/simplejson" + "github.com/grafana/grafana/pkg/tsdb" +) + +func (e *StackdriverExecutor) ensureDefaultProject(ctx context.Context, tsdbQuery *tsdb.TsdbQuery) (*tsdb.Response, error) { + queryResult := &tsdb.QueryResult{Meta: simplejson.New(), RefId: tsdbQuery.Queries[0].RefId} + result := &tsdb.Response{ + Results: make(map[string]*tsdb.QueryResult), + } + defaultProject, err := e.getDefaultProject(ctx) + if err != nil { + return nil, err + } + + e.dsInfo.JsonData.Set("defaultProject", defaultProject) + queryResult.Meta.Set("defaultProject", defaultProject) + result.Results[tsdbQuery.Queries[0].RefId] = queryResult + return result, nil +} diff --git a/pkg/tsdb/stackdriver/stackdriver.go b/pkg/tsdb/stackdriver/stackdriver.go index 96242dfdec4..8b903ba0113 100644 --- a/pkg/tsdb/stackdriver/stackdriver.go +++ b/pkg/tsdb/stackdriver/stackdriver.go @@ -16,6 +16,7 @@ import ( "time" "golang.org/x/net/context/ctxhttp" + "golang.org/x/oauth2/google" "github.com/grafana/grafana/pkg/api/pluginproxy" "github.com/grafana/grafana/pkg/components/null" @@ -34,6 +35,11 @@ var ( metricNameFormat *regexp.Regexp ) +const ( + gceAuthentication string = "gce" + jwtAuthentication string = "jwt" +) + // StackdriverExecutor executes queries for the Stackdriver datasource type StackdriverExecutor struct { httpClient *http.Client @@ -71,6 +77,8 @@ func (e *StackdriverExecutor) Query(ctx context.Context, dsInfo *models.DataSour switch queryType { case "annotationQuery": result, err = e.executeAnnotationQuery(ctx, tsdbQuery) + case "ensureDefaultProjectQuery": + result, err = e.ensureDefaultProject(ctx, tsdbQuery) case "timeSeriesQuery": fallthrough default: @@ -85,6 +93,16 @@ func (e *StackdriverExecutor) executeTimeSeriesQuery(ctx context.Context, tsdbQu Results: make(map[string]*tsdb.QueryResult), } + authenticationType := e.dsInfo.JsonData.Get("authenticationType").MustString(jwtAuthentication) + if authenticationType == gceAuthentication { + defaultProject, err := e.getDefaultProject(ctx) + if err != nil { + return nil, fmt.Errorf("Failed to retrieve default project from GCE metadata server. error: %v", err) + } + + e.dsInfo.JsonData.Set("defaultProject", defaultProject) + } + queries, err := e.buildQueries(tsdbQuery) if err != nil { return nil, err @@ -550,8 +568,6 @@ func (e *StackdriverExecutor) createRequest(ctx context.Context, dsInfo *models. if !ok { return nil, errors.New("Unable to find datasource plugin Stackdriver") } - projectName := dsInfo.JsonData.Get("defaultProject").MustString() - proxyPass := fmt.Sprintf("stackdriver%s", "v3/projects/"+projectName+"/timeSeries") var stackdriverRoute *plugins.AppPluginRoute for _, route := range plugin.Routes { @@ -561,7 +577,22 @@ func (e *StackdriverExecutor) createRequest(ctx context.Context, dsInfo *models. } } + projectName := dsInfo.JsonData.Get("defaultProject").MustString() + proxyPass := fmt.Sprintf("stackdriver%s", "v3/projects/"+projectName+"/timeSeries") + pluginproxy.ApplyRoute(ctx, req, proxyPass, stackdriverRoute, dsInfo) return req, nil } + +func (e *StackdriverExecutor) getDefaultProject(ctx context.Context) (string, error) { + authenticationType := e.dsInfo.JsonData.Get("authenticationType").MustString(jwtAuthentication) + if authenticationType == gceAuthentication { + defaultCredentials, err := google.FindDefaultCredentials(ctx, "https://www.googleapis.com/auth/monitoring.read") + if err != nil { + return "", fmt.Errorf("Failed to retrieve default project from GCE metadata server. error: %v", err) + } + return defaultCredentials.ProjectID, nil + } + return e.dsInfo.JsonData.Get("defaultProject").MustString(), nil +} diff --git a/public/app/plugins/datasource/stackdriver/config_ctrl.ts b/public/app/plugins/datasource/stackdriver/config_ctrl.ts index d3847562156..e74659536c8 100644 --- a/public/app/plugins/datasource/stackdriver/config_ctrl.ts +++ b/public/app/plugins/datasource/stackdriver/config_ctrl.ts @@ -5,13 +5,23 @@ export class StackdriverConfigCtrl { jsonText: string; validationErrors: string[] = []; inputDataValid: boolean; + authenticationTypes: any[]; + defaultAuthenticationType: string; /** @ngInject */ constructor(datasourceSrv) { + this.defaultAuthenticationType = 'jwt'; this.datasourceSrv = datasourceSrv; this.current.jsonData = this.current.jsonData || {}; + this.current.jsonData.authenticationType = this.current.jsonData.authenticationType + ? this.current.jsonData.authenticationType + : this.defaultAuthenticationType; this.current.secureJsonData = this.current.secureJsonData || {}; this.current.secureJsonFields = this.current.secureJsonFields || {}; + this.authenticationTypes = [ + { key: this.defaultAuthenticationType, value: 'Google JWT File' }, + { key: 'gce', value: 'GCE Default Service Account' }, + ]; } save(jwt) { @@ -35,6 +45,10 @@ export class StackdriverConfigCtrl { this.validationErrors.push('Client Email field missing in JWT file.'); } + if (!jwt.project_id || jwt.project_id.length === 0) { + this.validationErrors.push('Project Id field missing in JWT file.'); + } + if (this.validationErrors.length === 0) { this.inputDataValid = true; return true; @@ -67,7 +81,7 @@ export class StackdriverConfigCtrl { this.inputDataValid = false; this.jsonText = ''; - this.current.jsonData = {}; + this.current.jsonData = Object.assign({}, { authenticationType: this.current.jsonData.authenticationType }); this.current.secureJsonData = {}; this.current.secureJsonFields = {}; } diff --git a/public/app/plugins/datasource/stackdriver/datasource.ts b/public/app/plugins/datasource/stackdriver/datasource.ts index b77abdbdab3..cda952c23b9 100644 --- a/public/app/plugins/datasource/stackdriver/datasource.ts +++ b/public/app/plugins/datasource/stackdriver/datasource.ts @@ -1,11 +1,14 @@ import { stackdriverUnitMappings } from './constants'; import appEvents from 'app/core/app_events'; +import _ from 'lodash'; export default class StackdriverDatasource { id: number; url: string; baseUrl: string; projectName: string; + authenticationType: string; + queryPromise: Promise; /** @ngInject */ constructor(instanceSettings, private backendSrv, private templateSrv, private timeSrv) { @@ -14,6 +17,7 @@ export default class StackdriverDatasource { this.doRequest = this.doRequest; this.id = instanceSettings.id; this.projectName = instanceSettings.jsonData.defaultProject || ''; + this.authenticationType = instanceSettings.jsonData.authenticationType || 'jwt'; } async getTimeSeries(options) { @@ -46,16 +50,20 @@ export default class StackdriverDatasource { }; }); - const { data } = await this.backendSrv.datasourceRequest({ - url: '/api/tsdb/query', - method: 'POST', - data: { - from: options.range.from.valueOf().toString(), - to: options.range.to.valueOf().toString(), - queries, - }, - }); - return data; + if (queries.length > 0) { + const { data } = await this.backendSrv.datasourceRequest({ + url: '/api/tsdb/query', + method: 'POST', + data: { + from: options.range.from.valueOf().toString(), + to: options.range.to.valueOf().toString(), + queries, + }, + }); + return data; + } else { + return { results: [] }; + } } async getLabels(metricType, refId) { @@ -99,31 +107,34 @@ export default class StackdriverDatasource { } async query(options) { - const result = []; - const data = await this.getTimeSeries(options); - if (data.results) { - Object['values'](data.results).forEach(queryRes => { - if (!queryRes.series) { - return; - } - - const unit = this.resolvePanelUnitFromTargets(options.targets); - queryRes.series.forEach(series => { - let timeSerie: any = { - target: series.name, - datapoints: series.points, - refId: queryRes.refId, - meta: queryRes.meta, - }; - if (unit) { - timeSerie = { ...timeSerie, unit }; + this.queryPromise = new Promise(async resolve => { + const result = []; + const data = await this.getTimeSeries(options); + if (data.results) { + Object['values'](data.results).forEach(queryRes => { + if (!queryRes.series) { + return; } - result.push(timeSerie); + this.projectName = queryRes.meta.defaultProject; + const unit = this.resolvePanelUnitFromTargets(options.targets); + queryRes.series.forEach(series => { + let timeSerie: any = { + target: series.name, + datapoints: series.points, + refId: queryRes.refId, + meta: queryRes.meta, + }; + if (unit) { + timeSerie = { ...timeSerie, unit }; + } + result.push(timeSerie); + }); }); - }); - } + } - return { data: result }; + resolve({ data: result }); + }); + return this.queryPromise; } async annotationQuery(options) { @@ -173,76 +184,84 @@ export default class StackdriverDatasource { throw new Error('Template variables support is not yet imlemented'); } - testDatasource() { - const path = `v3/projects/${this.projectName}/metricDescriptors`; - return this.doRequest(`${this.baseUrl}${path}`) - .then(response => { - if (response.status === 200) { - return { - status: 'success', - message: 'Successfully queried the Stackdriver API.', - title: 'Success', - }; - } - - return { - status: 'error', - message: 'Returned http status code ' + response.status, - }; - }) - .catch(error => { - let message = 'Stackdriver: '; - message += error.statusText ? error.statusText + ': ' : ''; - + async testDatasource() { + let status, message; + const defaultErrorMessage = 'Cannot connect to Stackdriver API'; + try { + const projectName = await this.getDefaultProject(); + const path = `v3/projects/${projectName}/metricDescriptors`; + const response = await this.doRequest(`${this.baseUrl}${path}`); + if (response.status === 200) { + status = 'success'; + message = 'Successfully queried the Stackdriver API.'; + } else { + status = 'error'; + message = response.statusText ? response.statusText : defaultErrorMessage; + } + } catch (error) { + status = 'error'; + if (_.isString(error)) { + message = error; + } else { + message = 'Stackdriver: '; + message += error.statusText ? error.statusText : defaultErrorMessage; if (error.data && error.data.error && error.data.error.code) { - // 400, 401 - message += error.data.error.code + '. ' + error.data.error.message; - } else { - message += 'Cannot connect to Stackdriver API'; + message += ': ' + error.data.error.code + '. ' + error.data.error.message; } - return { - status: 'error', - message: message, - }; - }); + } + } finally { + return { + status, + message, + }; + } } - async getProjects() { - const response = await this.doRequest(`/cloudresourcemanager/v1/projects`); - return response.data.projects.map(p => ({ id: p.projectId, name: p.name })); + formatStackdriverError(error) { + let message = 'Stackdriver: '; + message += error.statusText ? error.statusText + ': ' : ''; + if (error.data && error.data.error) { + try { + const res = JSON.parse(error.data.error); + message += res.error.code + '. ' + res.error.message; + } catch (err) { + message += error.data.error; + } + } else { + message += 'Cannot connect to Stackdriver API'; + } + return message; } async getDefaultProject() { try { - const projects = await this.getProjects(); - if (projects && projects.length > 0) { - const test = projects.filter(p => p.id === this.projectName)[0]; - return test; + if (this.authenticationType === 'gce' || !this.projectName) { + const { data } = await this.backendSrv.datasourceRequest({ + url: '/api/tsdb/query', + method: 'POST', + data: { + queries: [ + { + refId: 'ensureDefaultProjectQuery', + type: 'ensureDefaultProjectQuery', + datasourceId: this.id, + }, + ], + }, + }); + this.projectName = data.results.ensureDefaultProjectQuery.meta.defaultProject; + return this.projectName; } else { - throw new Error('No projects found'); + return this.projectName; } } catch (error) { - let message = 'Projects cannot be fetched: '; - message += error.statusText ? error.statusText + ': ' : ''; - if (error && error.data && error.data.error && error.data.error.message) { - if (error.data.error.code === 403) { - message += ` - A list of projects could not be fetched from the Google Cloud Resource Manager API. - You might need to enable it first: - https://console.developers.google.com/apis/library/cloudresourcemanager.googleapis.com`; - } else { - message += error.data.error.code + '. ' + error.data.error.message; - } - } else { - message += 'Cannot connect to Stackdriver API'; - } - appEvents.emit('ds-request-error', message); + throw this.formatStackdriverError(error); } } - async getMetricTypes(projectId: string) { + async getMetricTypes(projectName: string) { try { - const metricsApiPath = `v3/projects/${projectId}/metricDescriptors`; + const metricsApiPath = `v3/projects/${projectName}/metricDescriptors`; const { data } = await this.doRequest(`${this.baseUrl}${metricsApiPath}`); const metrics = data.metricDescriptors.map(m => { @@ -256,7 +275,8 @@ export default class StackdriverDatasource { return metrics; } catch (error) { - console.log(error); + appEvents.emit('ds-request-error', this.formatStackdriverError(error)); + return []; } } diff --git a/public/app/plugins/datasource/stackdriver/partials/config.html b/public/app/plugins/datasource/stackdriver/partials/config.html index 46b79d8bb0d..9be7eef0e68 100644 --- a/public/app/plugins/datasource/stackdriver/partials/config.html +++ b/public/app/plugins/datasource/stackdriver/partials/config.html @@ -1,37 +1,54 @@
-
GCP Service Account
+

Stackdriver Authentication

+

There are two ways to authenticate the Stackdriver plugin - either by uploading a Service Account key file, or by + automatically retrieving credentials from the Google metadata server. The latter option is only available + when running Grafana on a GCE virtual machine.

+ +
Uploading a Service Account Key File

- To authenticate with the Stackdriver API, you need to create a Google Cloud Platform (GCP) Service Account for + First you need to create a Google Cloud Platform (GCP) Service Account for the Project you want to show data for. A Grafana datasource integrates with one GCP Project. If you want to visualize data from multiple GCP Projects then you need to create one datasource per GCP Project.

- The Monitoring Viewer role provides all the permissions that Grafana needs. + The Monitoring Viewer role provides all the permissions that Grafana needs. The following API + needs to be enabled on GCP for the datasource to work: Monitoring + API

+ +
GCE Default Service Account

- The following APIs need to be enabled on GCP for the datasource to work: -

+ If Grafana is running on a Google Compute Engine (GCE) virtual machine, it is possible for Grafana to + automatically retrieve the default project id and authentication token from the metadata server. In order for this to + work, you need to make sure that you have a service account that is setup as the default account for the virtual + machine and that the service account has been given read access to the Stackdriver Monitoring API.

+

Detailed instructions on how to create a Service Account can be found in - the documentation.

+ the documentation. +

-

Service Account Authentication

+

Authentication

Upload your Service Account key file or paste in the contents of the file. The file contents will be encrypted and saved in the Grafana database.
-
+
+
+ Authentication Type +
+ +
+
+
+ +
@@ -52,23 +69,23 @@
-
+
Uploaded Key Details
- Project + Project
- Client Email - -
+ Client Email + +
- Token URI + Token URI
- Private Key + Private Key
@@ -81,6 +98,8 @@
-
- Do not forget to save your changes after uploading a file. -
+

Do not forget to save your changes after uploading a file.

+ +

+ Verify GCE default service account by clicking Save & Test

diff --git a/public/app/plugins/datasource/stackdriver/partials/query.editor.html b/public/app/plugins/datasource/stackdriver/partials/query.editor.html index 7cc93e9d1a8..98c8fcc83e8 100755 --- a/public/app/plugins/datasource/stackdriver/partials/query.editor.html +++ b/public/app/plugins/datasource/stackdriver/partials/query.editor.html @@ -15,8 +15,7 @@
Project - +