SQL: Migrate to use SDK contracts (#36635)

* convert SQLs to use sdk contracts

* make draft

* postgres

* intermedia

* get datasourceinfo filled at the beginning of the service

* move the interval into package because of cyclict  import and fix all postgres tests

* fix mysql test

* fix mssql

* fix the test for pr https://github.com/grafana/grafana/issues/35839

* fix some issue about intervalv2 package

* update sql test

* wire migration for SQLs

* add sqls to the background process

* make it register instead of register and start

* revert formatting

* fix tests

* fix linter

* remove integration test

* Postgres test fix

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
This commit is contained in:
ying-jeanne
2021-09-07 09:35:37 +02:00
committed by GitHub
co-authored by Marcus Efraimsson
parent 51aeb717a4
commit c989f0fbbe
40 changed files with 1632 additions and 1495 deletions
@@ -51,7 +51,7 @@ These examples show how often and when reminders are sent for a triggered alert.
## List of supported notifiers
| Name | Type | Supports images | Support alert rule tags |
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- |
| --------------------------------------------- | ------------------------- | ------------------ | ----------------------- | --- |
| [DingDing](#dingdingdingtalk) | `dingding` | yes, external only | no |
| [Discord](#discord) | `discord` | yes | no |
| [Email](#email) | `email` | yes | no |
@@ -70,7 +70,7 @@ These examples show how often and when reminders are sent for a triggered alert.
| Telegram | `telegram` | yes | no |
| Threema | `threema` | yes, external only | no |
| VictorOps | `victorops` | yes, external only | yes |
| [Webhook](#webhook) | `webhook` | yes, external only | yes | |
| [Webhook](#webhook) | `webhook` | yes, external only | yes | |
### Email
@@ -258,7 +258,7 @@ Alertmanager handles alerts sent by client applications such as Prometheus serve
### Sensu Go
Grafana alert notifications can be sent to [Sensu]((https://sensu.io)) Go as events via the API. This operation requires an API key. For information on creating this key, refer to [Sensu Go documentation](https://docs.sensu.io/sensu-go/latest/operations/control-access/use-apikeys/#api-key-authentication).
Grafana alert notifications can be sent to [Sensu](<(https://sensu.io)>) Go as events via the API. This operation requires an API key. For information on creating this key, refer to [Sensu Go documentation](https://docs.sensu.io/sensu-go/latest/operations/control-access/use-apikeys/#api-key-authentication).
## Enable images in notifications {#external-image-store}
@@ -7,7 +7,7 @@ weight = 400
# Create a Cortex or Loki managed recording rule
You can create and manage recording rules for an external Cortex or Loki instance. Recording rules calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
You can create and manage recording rules for an external Cortex or Loki instance. Recording rules calculate frequently needed expressions or computationally expensive expressions in advance and save the result as a new set of time series. Querying this new time series is faster, especially for dashboards since they query the same expression every time the dashboards refresh.
For both Cortex and Loki data sources to work with Grafana 8.0 alerting, enable the ruler API by configuring their respective services. The `local` rule storage type (default for Loki data source), only supports viewing of rules. If you want to edit rules, then configure one of the other rule storage types.
@@ -34,7 +34,7 @@ This section describes the fields you fill out to create a recording rule.
### Rule type
- **Rule name -** Enter a descriptive name. The name will get displayed in the alert rule list. It will also get added as an `alertname` label to every alert instance that is created from this rule. Recording rules names must be valid [metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
- **Rule name -** Enter a descriptive name. The name will get displayed in the alert rule list. It will also get added as an `alertname` label to every alert instance that is created from this rule. Recording rules names must be valid [metric names](https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels).
- **Rule type -** Select **Cortex / Loki managed recording rule**.
- **Data source -** Select a Prometheus or Loki data source. Only data sources that support Cortex ruler API are available.
- **Namespace -** Select an existing rule namespace or click **Add new** and enter a name to create a new one. Namespaces can contain one or more rule groups and have only organizational purpose.
@@ -48,7 +48,6 @@ Enter a PromQL or LogQL expression. The result of this expression will get recor
![Query section](/static/img/docs/alerting/unified/rule-edit-cortex-recording-rule-query-8-2.png 'Query section screenshot')
### Details
You can optionally define labels in the details section.
@@ -75,11 +75,11 @@ See or [expressions documentation]({{< relref "../../../panels/expressions.md" >
Toggle **Configure no data and error handling** switch to configure how the rule should handle cases where evaluation results in error or returns no data.
| No Data Option | Description |
| -------------- | ----------------------------------------------------------------------------------------------------------- |
| No Data Option | Description |
| -------------- | ----------------------------------------------------------------------------------------------------- |
| No Data | Set alert state to `NoData` and rule state to `Normal` (notifications are not sent on NoData states). |
| Alerting | Set alert rule state to `Alerting`. |
| Ok | Set alert rule state to `Normal`. |
| Alerting | Set alert rule state to `Alerting`. |
| Ok | Set alert rule state to `Normal`. |
| Error or timeout option | Description |
| ----------------------- | ---------------------------------- |
@@ -15,7 +15,7 @@ A namespace contains one or more groups. To rename a namespace, find a group tha
1. Hover your cursor over the Alerting (bell) icon in the side menu.
1. Locate a group that belongs to the namespace you want to edit and click the edit (pen) icon.
1. Enter a new name in the **Namespace** field, then click **Save changes**.
1. Enter a new name in the **Namespace** field, then click **Save changes**.
A new namespace is created and all groups are copied into this namespace from the old one. The old namespace is deleted.
@@ -31,4 +31,4 @@ The rules within a group are run sequentially at a regular interval, the default
If you remaned the group, a new group is created that has all the rules from the old group, and the old group deleted.
![Group list](/static/img/docs/alerting/unified/rule-list-edit-cortex-loki-icon-8-2.png 'Rule group list screenshot')
![Group edit modal](/static/img/docs/alerting/unified/rule-list-cortex-loki-edit-ns-group-8-2.png 'Rule group edit modal screenshot')
![Group edit modal](/static/img/docs/alerting/unified/rule-list-cortex-loki-edit-ns-group-8-2.png 'Rule group edit modal screenshot')
+2 -2
View File
@@ -353,6 +353,6 @@ Refer to this [issue comment](https://github.com/grafana/grafana/issues/35534#is
As of Grafana v8.1, we no longer support unencrypted storage of passwords and basic auth passwords.
>**Note":** Since Grafana v6.2, new or updated data sources store passwords and basic auth passwords encrypted. See [upgrade note]({{< relref "#ensure-encryption-of-data-source-secrets" >}}) for more information. However, unencrypted passwords and basic auth passwords were also allowed.
> **Note":** Since Grafana v6.2, new or updated data sources store passwords and basic auth passwords encrypted. See [upgrade note]({{< relref "#ensure-encryption-of-data-source-secrets" >}}) for more information. However, unencrypted passwords and basic auth passwords were also allowed.
To migrate to encrypted storage, follow the instructions from the [v6.2 upgrade notes]({{< relref "#ensure-encryption-of-data-source-secrets" >}}). You can also use a `grafana-cli` command to migrate all of your data sources to use encrypted storage of secrets. See [migrate data and encrypt passwords]({{< relref "../administration/cli.md#migrate-data-and-encrypt-passwords" >}}) for further instructions.
@@ -26,7 +26,10 @@ import (
"github.com/grafana/grafana/pkg/tsdb/graphite"
"github.com/grafana/grafana/pkg/tsdb/influxdb"
"github.com/grafana/grafana/pkg/tsdb/loki"
"github.com/grafana/grafana/pkg/tsdb/mssql"
"github.com/grafana/grafana/pkg/tsdb/mysql"
"github.com/grafana/grafana/pkg/tsdb/opentsdb"
"github.com/grafana/grafana/pkg/tsdb/postgres"
"github.com/grafana/grafana/pkg/tsdb/prometheus"
"github.com/grafana/grafana/pkg/tsdb/tempo"
"github.com/grafana/grafana/pkg/tsdb/testdatasource"
@@ -43,6 +46,7 @@ func ProvideBackgroundServiceRegistry(
_ *azuremonitor.Service, _ *cloudwatch.CloudWatchService, _ *elasticsearch.Service, _ *graphite.Service,
_ *influxdb.Service, _ *loki.Service, _ *opentsdb.Service, _ *prometheus.Service, _ *tempo.Service,
_ *testdatasource.TestDataPlugin, _ *plugindashboards.Service, _ *dashboardsnapshots.Service,
_ *postgres.Service, _ *mysql.Service, _ *mssql.Service,
) *BackgroundServiceRegistry {
return NewBackgroundServiceRegistry(
+4
View File
@@ -60,6 +60,8 @@ import (
"github.com/grafana/grafana/pkg/tsdb/graphite"
"github.com/grafana/grafana/pkg/tsdb/influxdb"
"github.com/grafana/grafana/pkg/tsdb/loki"
"github.com/grafana/grafana/pkg/tsdb/mssql"
"github.com/grafana/grafana/pkg/tsdb/mysql"
"github.com/grafana/grafana/pkg/tsdb/opentsdb"
"github.com/grafana/grafana/pkg/tsdb/postgres"
"github.com/grafana/grafana/pkg/tsdb/prometheus"
@@ -95,6 +97,8 @@ var wireBasicSet = wire.NewSet(
cloudmonitoring.ProvideService,
azuremonitor.ProvideService,
postgres.ProvideService,
mysql.ProvideService,
mssql.ProvideService,
httpclientprovider.New,
wire.Bind(new(httpclient.Provider), new(*sdkhttpclient.Provider)),
serverlock.ProvideService,
+3 -3
View File
@@ -19,7 +19,7 @@ import (
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/infra/httpclient"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"golang.org/x/net/context/ctxhttp"
)
@@ -104,13 +104,13 @@ func (c *baseClientImpl) GetTimeField() string {
func (c *baseClientImpl) GetMinInterval(queryInterval string) (time.Duration, error) {
timeInterval := c.ds.TimeInterval
return tsdb.GetIntervalFrom(queryInterval, timeInterval, 0, 5*time.Second)
return intervalv2.GetIntervalFrom(queryInterval, timeInterval, 0, 5*time.Second)
}
type multiRequest struct {
header map[string]interface{}
body interface{}
interval tsdb.Interval
interval intervalv2.Interval
}
func (c *baseClientImpl) executeBatchRequest(uriPath, uriQuery string, requests []*multiRequest) (*response, error) {
+2 -2
View File
@@ -13,7 +13,7 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/infra/httpclient"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -319,7 +319,7 @@ func createMultisearchForTest(t *testing.T, c Client) (*MultiSearchRequest, erro
t.Helper()
msb := c.MultiSearch()
s := msb.Search(tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
s := msb.Search(intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
s.Agg().DateHistogram("2", "@timestamp", func(a *DateHistogramAgg, ab AggBuilder) {
a.Interval = "$__interval"
+2 -2
View File
@@ -5,7 +5,7 @@ import (
"net/http"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
)
type response struct {
@@ -32,7 +32,7 @@ type SearchDebugInfo struct {
// SearchRequest represents a search request
type SearchRequest struct {
Index string
Interval tsdb.Interval
Interval intervalv2.Interval
Size int
Sort map[string]interface{}
Query *Query
@@ -4,13 +4,13 @@ import (
"strings"
"github.com/Masterminds/semver"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
)
// SearchRequestBuilder represents a builder which can build a search request
type SearchRequestBuilder struct {
version *semver.Version
interval tsdb.Interval
interval intervalv2.Interval
index string
size int
sort map[string]interface{}
@@ -20,7 +20,7 @@ type SearchRequestBuilder struct {
}
// NewSearchRequestBuilder create a new search request builder
func NewSearchRequestBuilder(version *semver.Version, interval tsdb.Interval) *SearchRequestBuilder {
func NewSearchRequestBuilder(version *semver.Version, interval intervalv2.Interval) *SearchRequestBuilder {
builder := &SearchRequestBuilder{
version: version,
interval: interval,
@@ -129,7 +129,7 @@ func NewMultiSearchRequestBuilder(version *semver.Version) *MultiSearchRequestBu
}
// Search initiates and returns a new search request builder
func (m *MultiSearchRequestBuilder) Search(interval tsdb.Interval) *SearchRequestBuilder {
func (m *MultiSearchRequestBuilder) Search(interval intervalv2.Interval) *SearchRequestBuilder {
b := NewSearchRequestBuilder(m.version, interval)
m.requestBuilders = append(m.requestBuilders, b)
return b
@@ -7,7 +7,7 @@ import (
"github.com/Masterminds/semver"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
. "github.com/smartystreets/goconvey/convey"
)
@@ -16,7 +16,7 @@ func TestSearchRequest(t *testing.T) {
timeField := "@timestamp"
Convey("Given new search request builder for es version 5", func() {
version5, _ := semver.NewVersion("5.0.0")
b := NewSearchRequestBuilder(version5, tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
b := NewSearchRequestBuilder(version5, intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
Convey("When building search request", func() {
sr, err := b.Build()
@@ -392,7 +392,7 @@ func TestSearchRequest(t *testing.T) {
Convey("Given new search request builder for es version 2", func() {
version2, _ := semver.NewVersion("2.0.0")
b := NewSearchRequestBuilder(version2, tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
b := NewSearchRequestBuilder(version2, intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
Convey("When adding doc value field", func() {
b.AddDocValueField(timeField)
@@ -452,7 +452,7 @@ func TestMultiSearchRequest(t *testing.T) {
b := NewMultiSearchRequestBuilder(version2)
Convey("When adding one search request", func() {
b.Search(tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
b.Search(intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
Convey("When building search request should contain one search request", func() {
mr, err := b.Build()
@@ -462,8 +462,8 @@ func TestMultiSearchRequest(t *testing.T) {
})
Convey("When adding two search requests", func() {
b.Search(tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
b.Search(tsdb.Interval{Value: 15 * time.Second, Text: "15s"})
b.Search(intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
b.Search(intervalv2.Interval{Value: 15 * time.Second, Text: "15s"})
Convey("When building search request should contain two search requests", func() {
mr, err := b.Build()
+3 -3
View File
@@ -14,15 +14,15 @@ import (
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/tsdb"
es "github.com/grafana/grafana/pkg/tsdb/elasticsearch/client"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
)
var eslog = log.New("tsdb.elasticsearch")
type Service struct {
HTTPClientProvider httpclient.Provider
intervalCalculator tsdb.Calculator
intervalCalculator intervalv2.Calculator
im instancemgmt.InstanceManager
}
@@ -49,7 +49,7 @@ func newService(im instancemgmt.InstanceManager, httpClientProvider httpclient.P
return &Service{
im: im,
HTTPClientProvider: httpClientProvider,
intervalCalculator: tsdb.NewCalculator(),
intervalCalculator: intervalv2.NewCalculator(),
}
}
+4 -4
View File
@@ -9,18 +9,18 @@ import (
"github.com/Masterminds/semver"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/tsdb"
es "github.com/grafana/grafana/pkg/tsdb/elasticsearch/client"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
)
type timeSeriesQuery struct {
client es.Client
dataQueries []backend.DataQuery
intervalCalculator tsdb.Calculator
intervalCalculator intervalv2.Calculator
}
var newTimeSeriesQuery = func(client es.Client, dataQuery []backend.DataQuery,
intervalCalculator tsdb.Calculator) *timeSeriesQuery {
intervalCalculator intervalv2.Calculator) *timeSeriesQuery {
return &timeSeriesQuery{
client: client,
dataQueries: dataQuery,
@@ -70,7 +70,7 @@ func (e *timeSeriesQuery) processQuery(q *Query, ms *es.MultiSearchRequestBuilde
if err != nil {
return err
}
intrvl, err := e.intervalCalculator.Calculate(e.dataQueries[0].TimeRange, minInterval, tsdb.Min)
intrvl, err := e.intervalCalculator.Calculate(e.dataQueries[0].TimeRange, minInterval, intervalv2.Min)
if err != nil {
return err
}
@@ -8,8 +8,8 @@ import (
"github.com/Masterminds/semver"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/tsdb"
es "github.com/grafana/grafana/pkg/tsdb/elasticsearch/client"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -1150,7 +1150,7 @@ func executeTsdbQuery(c es.Client, body string, from, to time.Time, minInterval
},
},
}
query := newTimeSeriesQuery(c, dataRequest.Queries, tsdb.NewCalculator(tsdb.CalculatorOptions{MinInterval: minInterval}))
query := newTimeSeriesQuery(c, dataRequest.Queries, intervalv2.NewCalculator(intervalv2.CalculatorOptions{MinInterval: minInterval}))
return query.execute()
}
+2 -2
View File
@@ -8,7 +8,7 @@ import (
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
)
var (
@@ -29,7 +29,7 @@ func (query *Query) Build(queryContext *backend.QueryDataRequest) (string, error
res += query.renderTz()
}
intervalText := tsdb.FormatDuration(query.Interval)
intervalText := intervalv2.FormatDuration(query.Interval)
intervalMs := int64(query.Interval / time.Millisecond)
res = strings.ReplaceAll(res, "$timeFilter", query.renderTimeFilter(queryContext))
@@ -1,4 +1,4 @@
package tsdb
package intervalv2
import (
"testing"
@@ -1,4 +1,4 @@
package tsdb
package intervalv2
import (
"fmt"
@@ -7,7 +7,6 @@ import (
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/gtime"
"github.com/grafana/grafana/pkg/tsdb/interval"
)
@@ -101,6 +100,11 @@ func (ic *intervalCalculator) CalculateSafeInterval(timerange backend.TimeRange,
// queryInterval is the string representation of query interval (min interval), e.g. "10ms" or "10s".
// queryIntervalMS is a pre-calculated numeric representation of the query interval in milliseconds.
func GetIntervalFrom(dsInterval, queryInterval string, queryIntervalMS int64, defaultInterval time.Duration) (time.Duration, error) {
// Apparently we are setting default value of queryInterval to 0s now
if queryInterval == "0s" {
queryInterval = ""
}
if queryInterval == "" {
if queryIntervalMS != 0 {
return time.Duration(queryIntervalMS) * time.Millisecond, nil
@@ -121,7 +125,7 @@ func GetIntervalFrom(dsInterval, queryInterval string, queryIntervalMS int64, de
if isPureNum {
interval += "s"
}
parsedInterval, err := gtime.ParseDuration(interval)
parsedInterval, err := time.ParseDuration(interval)
if err != nil {
return time.Duration(0), err
}
+5 -5
View File
@@ -18,7 +18,7 @@ import (
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"github.com/grafana/loki/pkg/logcli/client"
"github.com/grafana/loki/pkg/loghttp"
"github.com/grafana/loki/pkg/logproto"
@@ -29,7 +29,7 @@ import (
)
type Service struct {
intervalCalculator tsdb.Calculator
intervalCalculator intervalv2.Calculator
im instancemgmt.InstanceManager
plog log.Logger
}
@@ -38,7 +38,7 @@ func ProvideService(httpClientProvider httpclient.Provider, manager backendplugi
im := datasource.NewInstanceManager(newInstanceSettings(httpClientProvider))
s := &Service{
im: im,
intervalCalculator: tsdb.NewCalculator(),
intervalCalculator: intervalv2.NewCalculator(),
plog: log.New("tsdb.loki"),
}
@@ -195,12 +195,12 @@ func (s *Service) parseQuery(dsInfo *datasourceInfo, queryContext *backend.Query
start := query.TimeRange.From
end := query.TimeRange.To
dsInterval, err := tsdb.GetIntervalFrom(dsInfo.TimeInterval, model.Interval, int64(model.IntervalMS), time.Second)
dsInterval, err := intervalv2.GetIntervalFrom(dsInfo.TimeInterval, model.Interval, int64(model.IntervalMS), time.Second)
if err != nil {
return nil, fmt.Errorf("failed to parse Interval: %v", err)
}
interval, err := s.intervalCalculator.Calculate(query.TimeRange, dsInterval, tsdb.Min)
interval, err := s.intervalCalculator.Calculate(query.TimeRange, dsInterval, intervalv2.Min)
if err != nil {
return nil, err
}
+7 -7
View File
@@ -8,7 +8,7 @@ import (
"github.com/google/go-cmp/cmp"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"github.com/grafana/loki/pkg/loghttp"
p "github.com/prometheus/common/model"
"github.com/stretchr/testify/require"
@@ -62,7 +62,7 @@ func TestLoki(t *testing.T) {
}
service := &Service{
intervalCalculator: mockCalculator{
interval: tsdb.Interval{
interval: intervalv2.Interval{
Value: time.Second * 30,
},
},
@@ -93,7 +93,7 @@ func TestLoki(t *testing.T) {
}
service := &Service{
intervalCalculator: mockCalculator{
interval: tsdb.Interval{
interval: intervalv2.Interval{
Value: time.Minute * 2,
},
},
@@ -105,7 +105,7 @@ func TestLoki(t *testing.T) {
service = &Service{
intervalCalculator: mockCalculator{
interval: tsdb.Interval{
interval: intervalv2.Interval{
Value: time.Second * 2,
},
},
@@ -175,13 +175,13 @@ func TestParseResponse(t *testing.T) {
}
type mockCalculator struct {
interval tsdb.Interval
interval intervalv2.Interval
}
func (m mockCalculator) Calculate(timerange backend.TimeRange, minInterval time.Duration, intervalMode tsdb.IntervalMode) (tsdb.Interval, error) {
func (m mockCalculator) Calculate(timerange backend.TimeRange, minInterval time.Duration, intervalMode intervalv2.IntervalMode) (intervalv2.Interval, error) {
return m.interval, nil
}
func (m mockCalculator) CalculateSafeInterval(timerange backend.TimeRange, resolution int64) tsdb.Interval {
func (m mockCalculator) CalculateSafeInterval(timerange backend.TimeRange, resolution int64) intervalv2.Interval {
return m.interval
}
+10 -10
View File
@@ -6,8 +6,8 @@ import (
"strings"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/gtime"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
@@ -22,7 +22,7 @@ func newMssqlMacroEngine() sqleng.SQLMacroEngine {
return &msSQLMacroEngine{SQLMacroEngineBase: sqleng.NewSQLMacroEngineBase()}
}
func (m *msSQLMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange plugins.DataTimeRange,
func (m *msSQLMacroEngine) Interpolate(query *backend.DataQuery, timeRange backend.TimeRange,
sql string) (string, error) {
// TODO: Return any error
rExp, _ := regexp.Compile(sExpr)
@@ -48,7 +48,7 @@ func (m *msSQLMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange plu
return sql, nil
}
func (m *msSQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query plugins.DataSubQuery, name string, args []string) (string, error) {
func (m *msSQLMacroEngine) evaluateMacro(timeRange backend.TimeRange, query *backend.DataQuery, name string, args []string) (string, error) {
switch name {
case "__time":
if len(args) == 0 {
@@ -65,11 +65,11 @@ func (m *msSQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s BETWEEN '%s' AND '%s'", args[0], timeRange.GetFromAsTimeUTC().Format(time.RFC3339), timeRange.GetToAsTimeUTC().Format(time.RFC3339)), nil
return fmt.Sprintf("%s BETWEEN '%s' AND '%s'", args[0], timeRange.From.UTC().Format(time.RFC3339), timeRange.To.UTC().Format(time.RFC3339)), nil
case "__timeFrom":
return fmt.Sprintf("'%s'", timeRange.GetFromAsTimeUTC().Format(time.RFC3339)), nil
return fmt.Sprintf("'%s'", timeRange.From.UTC().Format(time.RFC3339)), nil
case "__timeTo":
return fmt.Sprintf("'%s'", timeRange.GetToAsTimeUTC().Format(time.RFC3339)), nil
return fmt.Sprintf("'%s'", timeRange.To.UTC().Format(time.RFC3339)), nil
case "__timeGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval", name)
@@ -95,16 +95,16 @@ func (m *msSQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsSecondsEpoch(), args[0], timeRange.GetToAsSecondsEpoch()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().Unix(), args[0], timeRange.To.UTC().Unix()), nil
case "__unixEpochNanoFilter":
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsTimeUTC().UnixNano(), args[0], timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().UnixNano(), args[0], timeRange.To.UTC().UnixNano()), nil
case "__unixEpochNanoFrom":
return fmt.Sprintf("%d", timeRange.GetFromAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.From.UTC().UnixNano()), nil
case "__unixEpochNanoTo":
return fmt.Sprintf("%d", timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.To.UTC().UnixNano()), nil
case "__unixEpochGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval and optional fill value", name)
+34 -46
View File
@@ -2,14 +2,12 @@ package mssql
import (
"fmt"
"strconv"
"sync"
"testing"
"time"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana-plugin-sdk-go/backend"
. "github.com/smartystreets/goconvey/convey"
"github.com/stretchr/testify/require"
)
@@ -17,16 +15,16 @@ import (
func TestMacroEngine(t *testing.T) {
Convey("MacroEngine", t, func() {
engine := &msSQLMacroEngine{}
query := plugins.DataSubQuery{
Model: simplejson.New(),
query := &backend.DataQuery{
JSON: []byte("{}"),
}
dfltTimeRange := plugins.DataTimeRange{}
dfltTimeRange := backend.TimeRange{}
Convey("Given a time range between 2018-04-12 00:00 and 2018-04-12 00:05", func() {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", Now: to, To: "now"}
timeRange := backend.TimeRange{From: from, To: to}
Convey("interpolate __time function", func() {
sql, err := engine.Interpolate(query, dfltTimeRange, "select $__time(time_column)")
@@ -92,41 +90,26 @@ func TestMacroEngine(t *testing.T) {
Convey("interpolate __timeGroup function with fill (value = NULL)", func() {
_, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', NULL)")
fill := query.Model.Get("fill").MustBool()
fillMode := query.Model.Get("fillMode").MustString()
fillInterval := query.Model.Get("fillInterval").MustInt()
So(err, ShouldBeNil)
So(fill, ShouldBeTrue)
So(fillMode, ShouldEqual, "null")
So(fillInterval, ShouldEqual, 5*time.Minute.Seconds())
queryJson, err := query.JSON.MarshalJSON()
So(err, ShouldBeNil)
So(string(queryJson), ShouldEqual, `{"fill":true,"fillInterval":300,"fillMode":"null"}`)
})
Convey("interpolate __timeGroup function with fill (value = previous)", func() {
_, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', previous)")
fill := query.Model.Get("fill").MustBool()
fillMode := query.Model.Get("fillMode").MustString()
fillInterval := query.Model.Get("fillInterval").MustInt()
So(err, ShouldBeNil)
So(fill, ShouldBeTrue)
So(fillMode, ShouldEqual, "previous")
So(fillInterval, ShouldEqual, 5*time.Minute.Seconds())
queryJson, err := query.JSON.MarshalJSON()
So(err, ShouldBeNil)
So(string(queryJson), ShouldEqual, `{"fill":true,"fillInterval":300,"fillMode":"previous"}`)
})
Convey("interpolate __timeGroup function with fill (value = float)", func() {
_, err := engine.Interpolate(query, timeRange, "GROUP BY $__timeGroup(time_column,'5m', 1.5)")
fill := query.Model.Get("fill").MustBool()
fillValue := query.Model.Get("fillValue").MustFloat64()
fillInterval := query.Model.Get("fillInterval").MustInt()
So(err, ShouldBeNil)
So(fill, ShouldBeTrue)
So(fillValue, ShouldEqual, 1.5)
So(fillInterval, ShouldEqual, 5*time.Minute.Seconds())
queryJson, err := query.JSON.MarshalJSON()
So(err, ShouldBeNil)
So(string(queryJson), ShouldEqual, `{"fill":true,"fillInterval":300,"fillMode":"value","fillValue":1.5}`)
})
Convey("interpolate __unixEpochFilter function", func() {
@@ -170,9 +153,10 @@ func TestMacroEngine(t *testing.T) {
Convey("Given a time range between 1960-02-01 07:00 and 1965-02-03 08:00", func() {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1965, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10),
strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
Convey("interpolate __timeFilter function", func() {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -199,9 +183,10 @@ func TestMacroEngine(t *testing.T) {
Convey("Given a time range between 1960-02-01 07:00 and 1980-02-03 08:00", func() {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1980, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10),
strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
Convey("interpolate __timeFilter function", func() {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -229,27 +214,30 @@ func TestMacroEngine(t *testing.T) {
func TestMacroEngineConcurrency(t *testing.T) {
engine := newMssqlMacroEngine()
query1 := plugins.DataSubQuery{
Model: simplejson.New(),
query1 := backend.DataQuery{
JSON: []byte{},
}
query2 := plugins.DataSubQuery{
Model: simplejson.New(),
query2 := backend.DataQuery{
JSON: []byte{},
}
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", To: "now", Now: to}
timeRange := backend.TimeRange{
From: from,
To: to,
}
var wg sync.WaitGroup
wg.Add(2)
go func(query plugins.DataSubQuery) {
go func(query backend.DataQuery) {
defer wg.Done()
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
}(query1)
go func(query plugins.DataSubQuery) {
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
go func(query backend.DataQuery) {
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
defer wg.Done()
}(query2)
+94 -28
View File
@@ -1,6 +1,8 @@
package mssql
import (
"context"
"encoding/json"
"fmt"
"net/url"
"reflect"
@@ -8,43 +10,103 @@ import (
"strconv"
"strings"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/datasource"
"github.com/grafana/grafana-plugin-sdk-go/backend/instancemgmt"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/util"
mssql "github.com/denisenkom/go-mssqldb"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
var logger = log.New("tsdb.mssql")
//nolint: staticcheck // plugins.DataPlugin deprecated
func NewExecutor(datasource *models.DataSource) (plugins.DataPlugin, error) {
cnnstr, err := generateConnectionString(datasource)
type Service struct {
im instancemgmt.InstanceManager
}
func ProvideService(manager backendplugin.Manager) (*Service, error) {
s := &Service{
im: datasource.NewInstanceManager(newInstanceSettings()),
}
factory := coreplugin.New(backend.ServeOpts{
QueryDataHandler: s,
})
if err := manager.Register("mssql", factory); err != nil {
logger.Error("Failed to register plugin", "error", err)
}
return s, nil
}
func (s *Service) getDataSourceHandler(pluginCtx backend.PluginContext) (*sqleng.DataSourceHandler, error) {
i, err := s.im.Get(pluginCtx)
if err != nil {
return nil, err
}
// TODO: Don't use global
if setting.Env == setting.Dev {
logger.Debug("getEngine", "connection", cnnstr)
}
instance := i.(*sqleng.DataSourceHandler)
return instance, nil
}
config := sqleng.DataPluginConfiguration{
DriverName: "mssql",
ConnectionString: cnnstr,
Datasource: datasource,
MetricColumnTypes: []string{"VARCHAR", "CHAR", "NVARCHAR", "NCHAR"},
func (s *Service) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
dsHandler, err := s.getDataSourceHandler(req.PluginContext)
if err != nil {
return nil, err
}
return dsHandler.QueryData(ctx, req)
}
queryResultTransformer := mssqlQueryResultTransformer{
log: logger,
func newInstanceSettings() datasource.InstanceFactoryFunc {
return func(settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
jsonData := sqleng.JsonData{
MaxOpenConns: 0,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Encrypt: "false",
}
err := json.Unmarshal(settings.JSONData, &jsonData)
if err != nil {
return nil, fmt.Errorf("error reading settings: %w", err)
}
dsInfo := sqleng.DataSourceInfo{
JsonData: jsonData,
URL: settings.URL,
User: settings.User,
Database: settings.Database,
ID: settings.ID,
Updated: settings.Updated,
UID: settings.UID,
DecryptedSecureJSONData: settings.DecryptedSecureJSONData,
}
cnnstr, err := generateConnectionString(dsInfo)
if err != nil {
return nil, err
}
// TODO: Don't use global
if setting.Env == setting.Dev {
logger.Debug("getEngine", "connection", cnnstr)
}
config := sqleng.DataPluginConfiguration{
DriverName: "mssql",
ConnectionString: cnnstr,
DSInfo: dsInfo,
MetricColumnTypes: []string{"VARCHAR", "CHAR", "NVARCHAR", "NCHAR"},
}
queryResultTransformer := mssqlQueryResultTransformer{
log: logger,
}
return sqleng.NewQueryDataHandler(config, &queryResultTransformer, newMssqlMacroEngine(), logger)
}
return sqleng.NewDataPlugin(config, &queryResultTransformer, newMssqlMacroEngine(), logger)
}
// ParseURL tries to parse an MSSQL URL string into a URL object.
@@ -68,11 +130,11 @@ func ParseURL(u string) (*url.URL, error) {
}, nil
}
func generateConnectionString(dataSource *models.DataSource) (string, error) {
func generateConnectionString(dsInfo sqleng.DataSourceInfo) (string, error) {
const dfltPort = "0"
var addr util.NetworkAddress
if dataSource.Url != "" {
u, err := ParseURL(dataSource.Url)
if dsInfo.URL != "" {
u, err := ParseURL(dsInfo.URL)
if err != nil {
return "", err
}
@@ -88,26 +150,30 @@ func generateConnectionString(dataSource *models.DataSource) (string, error) {
}
args := []interface{}{
"url", dataSource.Url, "host", addr.Host,
"url", dsInfo.URL, "host", addr.Host,
}
if addr.Port != "0" {
args = append(args, "port", addr.Port)
}
logger.Debug("Generating connection string", args...)
encrypt := dataSource.JsonData.Get("encrypt").MustString("false")
connStr := fmt.Sprintf("server=%s;database=%s;user id=%s;password=%s;",
addr.Host,
dataSource.Database,
dataSource.User,
dataSource.DecryptedPassword(),
dsInfo.Database,
dsInfo.User,
dsInfo.DecryptedSecureJSONData["password"],
)
// Port number 0 means to determine the port automatically, so we can let the driver choose
if addr.Port != "0" {
connStr += fmt.Sprintf("port=%s;", addr.Port)
}
if encrypt != "false" {
connStr += fmt.Sprintf("encrypt=%s;", encrypt)
if dsInfo.JsonData.Encrypt == "" {
dsInfo.JsonData.Encrypt = "false"
}
if dsInfo.JsonData.Encrypt != "false" {
connStr += fmt.Sprintf("encrypt=%s;", dsInfo.JsonData.Encrypt)
}
return connStr, nil
}
File diff suppressed because it is too large Load Diff
+10 -10
View File
@@ -6,9 +6,9 @@ import (
"regexp"
"strings"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/gtime"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
@@ -26,7 +26,7 @@ func newMysqlMacroEngine(logger log.Logger) sqleng.SQLMacroEngine {
return &mySQLMacroEngine{SQLMacroEngineBase: sqleng.NewSQLMacroEngineBase(), logger: logger}
}
func (m *mySQLMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange plugins.DataTimeRange, sql string) (string, error) {
func (m *mySQLMacroEngine) Interpolate(query *backend.DataQuery, timeRange backend.TimeRange, sql string) (string, error) {
matches := restrictedRegExp.FindAllStringSubmatch(sql, 1)
if len(matches) > 0 {
m.logger.Error("show grants, session_user(), current_user(), system_user() or user() not allowed in query")
@@ -57,7 +57,7 @@ func (m *mySQLMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange plu
return sql, nil
}
func (m *mySQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query plugins.DataSubQuery, name string, args []string) (string, error) {
func (m *mySQLMacroEngine) evaluateMacro(timeRange backend.TimeRange, query *backend.DataQuery, name string, args []string) (string, error) {
switch name {
case "__timeEpoch", "__time":
if len(args) == 0 {
@@ -69,11 +69,11 @@ func (m *mySQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s BETWEEN FROM_UNIXTIME(%d) AND FROM_UNIXTIME(%d)", args[0], timeRange.GetFromAsSecondsEpoch(), timeRange.GetToAsSecondsEpoch()), nil
return fmt.Sprintf("%s BETWEEN FROM_UNIXTIME(%d) AND FROM_UNIXTIME(%d)", args[0], timeRange.From.UTC().Unix(), timeRange.To.UTC().Unix()), nil
case "__timeFrom":
return fmt.Sprintf("FROM_UNIXTIME(%d)", timeRange.GetFromAsSecondsEpoch()), nil
return fmt.Sprintf("FROM_UNIXTIME(%d)", timeRange.From.UTC().Unix()), nil
case "__timeTo":
return fmt.Sprintf("FROM_UNIXTIME(%d)", timeRange.GetToAsSecondsEpoch()), nil
return fmt.Sprintf("FROM_UNIXTIME(%d)", timeRange.To.UTC().Unix()), nil
case "__timeGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval", name)
@@ -99,16 +99,16 @@ func (m *mySQLMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsSecondsEpoch(), args[0], timeRange.GetToAsSecondsEpoch()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().Unix(), args[0], timeRange.To.UTC().Unix()), nil
case "__unixEpochNanoFilter":
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsTimeUTC().UnixNano(), args[0], timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().UnixNano(), args[0], timeRange.To.UTC().UnixNano()), nil
case "__unixEpochNanoFrom":
return fmt.Sprintf("%d", timeRange.GetFromAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.From.UTC().UnixNano()), nil
case "__unixEpochNanoTo":
return fmt.Sprintf("%d", timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.To.UTC().UnixNano()), nil
case "__unixEpochGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval and optional fill value", name)
+21 -19
View File
@@ -2,14 +2,12 @@ package mysql
import (
"fmt"
"strconv"
"sync"
"testing"
"time"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/plugins"
. "github.com/smartystreets/goconvey/convey"
"github.com/stretchr/testify/require"
)
@@ -19,12 +17,12 @@ func TestMacroEngine(t *testing.T) {
engine := &mySQLMacroEngine{
logger: log.New("test"),
}
query := plugins.DataSubQuery{}
query := &backend.DataQuery{}
Convey("Given a time range between 2018-04-12 00:00 and 2018-04-12 00:05", func() {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", Now: to, To: "now"}
timeRange := backend.TimeRange{From: from, To: to}
Convey("interpolate __time function", func() {
sql, err := engine.Interpolate(query, timeRange, "select $__time(time_column)")
@@ -123,8 +121,10 @@ func TestMacroEngine(t *testing.T) {
Convey("Given a time range between 1960-02-01 07:00 and 1965-02-03 08:00", func() {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1965, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10), strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
Convey("interpolate __timeFilter function", func() {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -144,8 +144,10 @@ func TestMacroEngine(t *testing.T) {
Convey("Given a time range between 1960-02-01 07:00 and 1980-02-03 08:00", func() {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1980, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10), strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
Convey("interpolate __timeFilter function", func() {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -185,7 +187,7 @@ func TestMacroEngine(t *testing.T) {
}
for _, tc := range tcs {
_, err := engine.Interpolate(plugins.DataSubQuery{}, plugins.DataTimeRange{}, tc)
_, err := engine.Interpolate(&backend.DataQuery{}, backend.TimeRange{}, tc)
So(err.Error(), ShouldEqual, "invalid query - inspect Grafana server log for details")
}
})
@@ -194,27 +196,27 @@ func TestMacroEngine(t *testing.T) {
func TestMacroEngineConcurrency(t *testing.T) {
engine := newMysqlMacroEngine(log.New("test"))
query1 := plugins.DataSubQuery{
Model: simplejson.New(),
query1 := backend.DataQuery{
JSON: []byte{},
}
query2 := plugins.DataSubQuery{
Model: simplejson.New(),
query2 := backend.DataQuery{
JSON: []byte{},
}
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", To: "now", Now: to}
timeRange := backend.TimeRange{From: from, To: to}
var wg sync.WaitGroup
wg.Add(2)
go func(query plugins.DataSubQuery) {
go func(query backend.DataQuery) {
defer wg.Done()
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
}(query1)
go func(query plugins.DataSubQuery) {
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
go func(query backend.DataQuery) {
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
defer wg.Done()
}(query2)
+84 -21
View File
@@ -1,6 +1,8 @@
package mysql
import (
"context"
"encoding/json"
"errors"
"fmt"
"net/url"
@@ -10,15 +12,18 @@ import (
"time"
"github.com/VividCortex/mysqlerr"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/datasource"
"github.com/grafana/grafana-plugin-sdk-go/backend/instancemgmt"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/infra/httpclient"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/setting"
"github.com/go-sql-driver/mysql"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
@@ -28,47 +33,88 @@ const (
dateTimeFormat2 = "2006-01-02T15:04:05Z"
)
var logger = log.New("tsdb.mysql")
type Service struct {
Cfg *setting.Cfg
im instancemgmt.InstanceManager
}
func characterEscape(s string, escapeChar string) string {
return strings.ReplaceAll(s, escapeChar, url.QueryEscape(escapeChar))
}
//nolint: staticcheck // plugins.DataPlugin deprecated
func New(httpClientProvider httpclient.Provider) func(datasource *models.DataSource) (plugins.DataPlugin, error) {
//nolint: staticcheck // plugins.DataPlugin deprecated
return func(datasource *models.DataSource) (plugins.DataPlugin, error) {
logger := log.New("tsdb.mysql")
func ProvideService(cfg *setting.Cfg, manager backendplugin.Manager, httpClientProvider httpclient.Provider) (*Service, error) {
s := &Service{
Cfg: cfg,
im: datasource.NewInstanceManager(newInstanceSettings(httpClientProvider)),
}
factory := coreplugin.New(backend.ServeOpts{
QueryDataHandler: s,
})
if err := manager.Register("mysql", factory); err != nil {
logger.Error("Failed to register plugin", "error", err)
}
return s, nil
}
func newInstanceSettings(httpClientProvider httpclient.Provider) datasource.InstanceFactoryFunc {
return func(settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
jsonData := sqleng.JsonData{
MaxOpenConns: 0,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
}
err := json.Unmarshal(settings.JSONData, &jsonData)
if err != nil {
return nil, fmt.Errorf("error reading settings: %w", err)
}
dsInfo := sqleng.DataSourceInfo{
JsonData: jsonData,
URL: settings.URL,
User: settings.User,
Database: settings.Database,
ID: settings.ID,
Updated: settings.Updated,
UID: settings.UID,
DecryptedSecureJSONData: settings.DecryptedSecureJSONData,
}
protocol := "tcp"
if strings.HasPrefix(datasource.Url, "/") {
if strings.HasPrefix(dsInfo.URL, "/") {
protocol = "unix"
}
cnnstr := fmt.Sprintf("%s:%s@%s(%s)/%s?collation=utf8mb4_unicode_ci&parseTime=true&loc=UTC&allowNativePasswords=true",
characterEscape(datasource.User, ":"),
datasource.DecryptedPassword(),
characterEscape(dsInfo.User, ":"),
dsInfo.DecryptedSecureJSONData["password"],
protocol,
characterEscape(datasource.Url, ")"),
characterEscape(datasource.Database, "?"),
characterEscape(dsInfo.URL, ")"),
characterEscape(dsInfo.Database, "?"),
)
tlsConfig, err := datasource.GetTLSConfig(httpClientProvider)
opts, err := settings.HTTPClientOptions()
if err != nil {
return nil, err
}
tlsConfig, err := httpClientProvider.GetTLSConfig(opts)
if err != nil {
return nil, err
}
if tlsConfig.RootCAs != nil || len(tlsConfig.Certificates) > 0 {
tlsConfigString := fmt.Sprintf("ds%d", datasource.Id)
tlsConfigString := fmt.Sprintf("ds%d", settings.ID)
if err := mysql.RegisterTLSConfig(tlsConfigString, tlsConfig); err != nil {
return nil, err
}
cnnstr += "&tls=" + tlsConfigString
}
if datasource.JsonData != nil {
timezone, hasTimezone := datasource.JsonData.CheckGet("timezone")
if hasTimezone && timezone.MustString() != "" {
cnnstr += fmt.Sprintf("&time_zone='%s'", url.QueryEscape(timezone.MustString()))
}
if dsInfo.JsonData.Timezone != "" {
cnnstr += fmt.Sprintf("&time_zone='%s'", url.QueryEscape(dsInfo.JsonData.Timezone))
}
if setting.Env == setting.Dev {
@@ -78,7 +124,7 @@ func New(httpClientProvider httpclient.Provider) func(datasource *models.DataSou
config := sqleng.DataPluginConfiguration{
DriverName: "mysql",
ConnectionString: cnnstr,
Datasource: datasource,
DSInfo: dsInfo,
TimeColumnNames: []string{"time", "time_sec"},
MetricColumnTypes: []string{"CHAR", "VARCHAR", "TINYTEXT", "TEXT", "MEDIUMTEXT", "LONGTEXT"},
}
@@ -87,10 +133,27 @@ func New(httpClientProvider httpclient.Provider) func(datasource *models.DataSou
log: logger,
}
return sqleng.NewDataPlugin(config, &rowTransformer, newMysqlMacroEngine(logger), logger)
return sqleng.NewQueryDataHandler(config, &rowTransformer, newMysqlMacroEngine(logger), logger)
}
}
func (s *Service) getDataSourceHandler(pluginCtx backend.PluginContext) (*sqleng.DataSourceHandler, error) {
i, err := s.im.Get(pluginCtx)
if err != nil {
return nil, err
}
instance := i.(*sqleng.DataSourceHandler)
return instance, nil
}
func (s *Service) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
dsHandler, err := s.getDataSourceHandler(req.PluginContext)
if err != nil {
return nil, err
}
return dsHandler.QueryData(ctx, req)
}
type mysqlQueryResultTransformer struct {
log log.Logger
}
File diff suppressed because it is too large Load Diff
+10 -11
View File
@@ -6,8 +6,8 @@ import (
"strings"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/components/gtime"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
@@ -26,8 +26,7 @@ func newPostgresMacroEngine(timescaledb bool) sqleng.SQLMacroEngine {
}
}
func (m *postgresMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange plugins.DataTimeRange,
sql string) (string, error) {
func (m *postgresMacroEngine) Interpolate(query *backend.DataQuery, timeRange backend.TimeRange, sql string) (string, error) {
// TODO: Handle error
rExp, _ := regexp.Compile(sExpr)
var macroError error
@@ -67,7 +66,7 @@ func (m *postgresMacroEngine) Interpolate(query plugins.DataSubQuery, timeRange
}
//nolint: gocyclo
func (m *postgresMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, query plugins.DataSubQuery, name string, args []string) (string, error) {
func (m *postgresMacroEngine) evaluateMacro(timeRange backend.TimeRange, query *backend.DataQuery, name string, args []string) (string, error) {
switch name {
case "__time":
if len(args) == 0 {
@@ -84,11 +83,11 @@ func (m *postgresMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, que
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s BETWEEN '%s' AND '%s'", args[0], timeRange.GetFromAsTimeUTC().Format(time.RFC3339Nano), timeRange.GetToAsTimeUTC().Format(time.RFC3339Nano)), nil
return fmt.Sprintf("%s BETWEEN '%s' AND '%s'", args[0], timeRange.From.UTC().Format(time.RFC3339Nano), timeRange.To.UTC().Format(time.RFC3339Nano)), nil
case "__timeFrom":
return fmt.Sprintf("'%s'", timeRange.GetFromAsTimeUTC().Format(time.RFC3339Nano)), nil
return fmt.Sprintf("'%s'", timeRange.From.UTC().Format(time.RFC3339Nano)), nil
case "__timeTo":
return fmt.Sprintf("'%s'", timeRange.GetToAsTimeUTC().Format(time.RFC3339Nano)), nil
return fmt.Sprintf("'%s'", timeRange.To.UTC().Format(time.RFC3339Nano)), nil
case "__timeGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval and optional fill value", name)
@@ -123,16 +122,16 @@ func (m *postgresMacroEngine) evaluateMacro(timeRange plugins.DataTimeRange, que
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsSecondsEpoch(), args[0], timeRange.GetToAsSecondsEpoch()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().Unix(), args[0], timeRange.To.UTC().Unix()), nil
case "__unixEpochNanoFilter":
if len(args) == 0 {
return "", fmt.Errorf("missing time column argument for macro %v", name)
}
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.GetFromAsTimeUTC().UnixNano(), args[0], timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%s >= %d AND %s <= %d", args[0], timeRange.From.UTC().UnixNano(), args[0], timeRange.To.UTC().UnixNano()), nil
case "__unixEpochNanoFrom":
return fmt.Sprintf("%d", timeRange.GetFromAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.From.UTC().UnixNano()), nil
case "__unixEpochNanoTo":
return fmt.Sprintf("%d", timeRange.GetToAsTimeUTC().UnixNano()), nil
return fmt.Sprintf("%d", timeRange.To.UTC().UnixNano()), nil
case "__unixEpochGroup":
if len(args) < 2 {
return "", fmt.Errorf("macro %v needs time column and interval and optional fill value", name)
+24 -23
View File
@@ -2,13 +2,11 @@ package postgres
import (
"fmt"
"strconv"
"sync"
"testing"
"time"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/stretchr/testify/require"
)
@@ -17,12 +15,12 @@ func TestMacroEngine(t *testing.T) {
engine := newPostgresMacroEngine(timescaledbEnabled)
timescaledbEnabled = true
engineTS := newPostgresMacroEngine(timescaledbEnabled)
query := plugins.DataSubQuery{}
query := &backend.DataQuery{}
t.Run("Given a time range between 2018-04-12 00:00 and 2018-04-12 00:05", func(t *testing.T) {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", To: "now", Now: to}
timeRange := backend.TimeRange{From: from, To: to}
t.Run("interpolate __time function", func(t *testing.T) {
sql, err := engine.Interpolate(query, timeRange, "select $__time(time_column)")
@@ -151,9 +149,10 @@ func TestMacroEngine(t *testing.T) {
t.Run("Given a time range between 1960-02-01 07:00 and 1965-02-03 08:00", func(t *testing.T) {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1965, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10),
strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
t.Run("interpolate __timeFilter function", func(t *testing.T) {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -177,9 +176,10 @@ func TestMacroEngine(t *testing.T) {
t.Run("Given a time range between 1960-02-01 07:00 and 1980-02-03 08:00", func(t *testing.T) {
from := time.Date(1960, 2, 1, 7, 0, 0, 0, time.UTC)
to := time.Date(1980, 2, 3, 8, 0, 0, 0, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10),
strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
t.Run("interpolate __timeFilter function", func(t *testing.T) {
sql, err := engine.Interpolate(query, timeRange, "WHERE $__timeFilter(time_column)")
@@ -203,9 +203,10 @@ func TestMacroEngine(t *testing.T) {
t.Run("Given a time range between 1960-02-01 07:00:00.5 and 1980-02-03 08:00:00.5", func(t *testing.T) {
from := time.Date(1960, 2, 1, 7, 0, 0, 500e6, time.UTC)
to := time.Date(1980, 2, 3, 8, 0, 0, 500e6, time.UTC)
timeRange := plugins.NewDataTimeRange(
strconv.FormatInt(from.UnixNano()/int64(time.Millisecond), 10), strconv.FormatInt(to.UnixNano()/int64(time.Millisecond), 10))
timeRange := backend.TimeRange{
From: from,
To: to,
}
require.Equal(t, "1960-02-01T07:00:00.5Z", from.Format(time.RFC3339Nano))
require.Equal(t, "1980-02-03T08:00:00.5Z", to.Format(time.RFC3339Nano))
@@ -219,27 +220,27 @@ func TestMacroEngine(t *testing.T) {
func TestMacroEngineConcurrency(t *testing.T) {
engine := newPostgresMacroEngine(false)
query1 := plugins.DataSubQuery{
Model: simplejson.New(),
query1 := backend.DataQuery{
JSON: []byte{},
}
query2 := plugins.DataSubQuery{
Model: simplejson.New(),
query2 := backend.DataQuery{
JSON: []byte{},
}
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", To: "now", Now: to}
timeRange := backend.TimeRange{From: from, To: to}
var wg sync.WaitGroup
wg.Add(2)
go func(query plugins.DataSubQuery) {
go func(query backend.DataQuery) {
defer wg.Done()
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
}(query1)
go func(query plugins.DataSubQuery) {
_, err := engine.Interpolate(query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
go func(query backend.DataQuery) {
_, err := engine.Interpolate(&query, timeRange, "SELECT $__timeGroup(time_column,'5m')")
require.NoError(t, err)
defer wg.Done()
}(query2)
+99 -47
View File
@@ -1,72 +1,124 @@
package postgres
import (
"context"
"encoding/json"
"fmt"
"reflect"
"strconv"
"strings"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/datasource"
"github.com/grafana/grafana-plugin-sdk-go/backend/instancemgmt"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/util/errutil"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
func ProvideService(cfg *setting.Cfg) *PostgresService {
logger := log.New("tsdb.postgres")
return &PostgresService{
var logger = log.New("tsdb.postgres")
func ProvideService(cfg *setting.Cfg, manager backendplugin.Manager) (*Service, error) {
s := &Service{
Cfg: cfg,
logger: logger,
tlsManager: newTLSManager(logger, cfg.DataPath),
}
s.im = datasource.NewInstanceManager(s.newInstanceSettings())
factory := coreplugin.New(backend.ServeOpts{
QueryDataHandler: s,
})
if err := manager.Register("postgres", factory); err != nil {
logger.Error("Failed to register plugin", "error", err)
}
return s, nil
}
type PostgresService struct {
type Service struct {
Cfg *setting.Cfg
logger log.Logger
tlsManager tlsSettingsProvider
im instancemgmt.InstanceManager
}
//nolint: staticcheck // plugins.DataPlugin deprecated
func (s *PostgresService) NewExecutor(datasource *models.DataSource) (plugins.DataPlugin, error) {
s.logger.Debug("Creating Postgres query endpoint")
cnnstr, err := s.generateConnectionString(datasource)
func (s *Service) getDSInfo(pluginCtx backend.PluginContext) (*sqleng.DataSourceHandler, error) {
i, err := s.im.Get(pluginCtx)
if err != nil {
return nil, err
}
instance := i.(*sqleng.DataSourceHandler)
return instance, nil
}
if s.Cfg.Env == setting.Dev {
s.logger.Debug("getEngine", "connection", cnnstr)
}
config := sqleng.DataPluginConfiguration{
DriverName: "postgres",
ConnectionString: cnnstr,
Datasource: datasource,
MetricColumnTypes: []string{"UNKNOWN", "TEXT", "VARCHAR", "CHAR"},
}
queryResultTransformer := postgresQueryResultTransformer{
log: s.logger,
}
timescaledb := datasource.JsonData.Get("timescaledb").MustBool(false)
plugin, err := sqleng.NewDataPlugin(config, &queryResultTransformer, newPostgresMacroEngine(timescaledb),
s.logger)
func (s *Service) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
dsInfo, err := s.getDSInfo(req.PluginContext)
if err != nil {
s.logger.Error("Failed connecting to Postgres", "err", err)
return nil, err
}
return dsInfo.QueryData(ctx, req)
}
s.logger.Debug("Successfully connected to Postgres")
return plugin, nil
func (s *Service) newInstanceSettings() datasource.InstanceFactoryFunc {
return func(settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
logger.Debug("Creating Postgres query endpoint")
jsonData := sqleng.JsonData{
MaxOpenConns: 0,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Timescaledb: false,
ConfigurationMethod: "file-path",
}
err := json.Unmarshal(settings.JSONData, &jsonData)
if err != nil {
return nil, fmt.Errorf("error reading settings: %w", err)
}
dsInfo := sqleng.DataSourceInfo{
JsonData: jsonData,
URL: settings.URL,
User: settings.User,
Database: settings.Database,
ID: settings.ID,
Updated: settings.Updated,
UID: settings.UID,
DecryptedSecureJSONData: settings.DecryptedSecureJSONData,
}
cnnstr, err := s.generateConnectionString(dsInfo)
if err != nil {
return nil, err
}
if s.Cfg.Env == setting.Dev {
logger.Debug("getEngine", "connection", cnnstr)
}
config := sqleng.DataPluginConfiguration{
DriverName: "postgres",
ConnectionString: cnnstr,
DSInfo: dsInfo,
MetricColumnTypes: []string{"UNKNOWN", "TEXT", "VARCHAR", "CHAR"},
}
queryResultTransformer := postgresQueryResultTransformer{
log: logger,
}
handler, err := sqleng.NewQueryDataHandler(config, &queryResultTransformer, newPostgresMacroEngine(dsInfo.JsonData.Timescaledb),
logger)
if err != nil {
logger.Error("Failed connecting to Postgres", "err", err)
return nil, err
}
logger.Debug("Successfully connected to Postgres")
return handler, nil
}
}
// escape single quotes and backslashes in Postgres connection string parameters.
@@ -74,14 +126,14 @@ func escape(input string) string {
return strings.ReplaceAll(strings.ReplaceAll(input, `\`, `\\`), "'", `\'`)
}
func (s *PostgresService) generateConnectionString(datasource *models.DataSource) (string, error) {
func (s *Service) generateConnectionString(dsInfo sqleng.DataSourceInfo) (string, error) {
var host string
var port int
if strings.HasPrefix(datasource.Url, "/") {
host = datasource.Url
s.logger.Debug("Generating connection string with Unix socket specifier", "socket", host)
if strings.HasPrefix(dsInfo.URL, "/") {
host = dsInfo.URL
logger.Debug("Generating connection string with Unix socket specifier", "socket", host)
} else {
sp := strings.SplitN(datasource.Url, ":", 2)
sp := strings.SplitN(dsInfo.URL, ":", 2)
host = sp[0]
if len(sp) > 1 {
var err error
@@ -90,19 +142,19 @@ func (s *PostgresService) generateConnectionString(datasource *models.DataSource
return "", errutil.Wrapf(err, "invalid port in host specifier %q", sp[1])
}
s.logger.Debug("Generating connection string with network host/port pair", "host", host, "port", port)
logger.Debug("Generating connection string with network host/port pair", "host", host, "port", port)
} else {
s.logger.Debug("Generating connection string with network host", "host", host)
logger.Debug("Generating connection string with network host", "host", host)
}
}
connStr := fmt.Sprintf("user='%s' password='%s' host='%s' dbname='%s'",
escape(datasource.User), escape(datasource.DecryptedPassword()), escape(host), escape(datasource.Database))
escape(dsInfo.User), escape(dsInfo.DecryptedSecureJSONData["password"]), escape(host), escape(dsInfo.Database))
if port > 0 {
connStr += fmt.Sprintf(" port=%d", port)
}
tlsSettings, err := s.tlsManager.getTLSSettings(datasource)
tlsSettings, err := s.tlsManager.getTLSSettings(dsInfo)
if err != nil {
return "", err
}
@@ -111,19 +163,19 @@ func (s *PostgresService) generateConnectionString(datasource *models.DataSource
// Attach root certificate if provided
if tlsSettings.RootCertFile != "" {
s.logger.Debug("Setting server root certificate", "tlsRootCert", tlsSettings.RootCertFile)
logger.Debug("Setting server root certificate", "tlsRootCert", tlsSettings.RootCertFile)
connStr += fmt.Sprintf(" sslrootcert='%s'", escape(tlsSettings.RootCertFile))
}
// Attach client certificate and key if both are provided
if tlsSettings.CertFile != "" && tlsSettings.CertKeyFile != "" {
s.logger.Debug("Setting TLS/SSL client auth", "tlsCert", tlsSettings.CertFile, "tlsKey", tlsSettings.CertKeyFile)
logger.Debug("Setting TLS/SSL client auth", "tlsCert", tlsSettings.CertFile, "tlsKey", tlsSettings.CertKeyFile)
connStr += fmt.Sprintf(" sslcert='%s' sslkey='%s'", escape(tlsSettings.CertFile), escape(tlsSettings.CertKeyFile))
} else if tlsSettings.CertFile != "" || tlsSettings.CertKeyFile != "" {
return "", fmt.Errorf("TLS/SSL client certificate and key must both be specified")
}
s.logger.Debug("Generated Postgres connection string successfully")
logger.Debug("Generated Postgres connection string successfully")
return connStr, nil
}
File diff suppressed because it is too large Load Diff
+44 -47
View File
@@ -8,17 +8,26 @@ import (
"strconv"
"strings"
"sync"
"time"
"github.com/grafana/grafana/pkg/infra/fs"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
)
var validateCertFunc = validateCertFilePaths
var writeCertFileFunc = writeCertFile
type certFileType int
const (
rootCert = iota
clientCert
clientKey
)
type tlsSettingsProvider interface {
getTLSSettings(datasource *models.DataSource) (tlsSettings, error)
getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error)
}
type datasourceCacheManager struct {
@@ -48,46 +57,37 @@ type tlsSettings struct {
CertKeyFile string
}
func (m *tlsManager) getTLSSettings(datasource *models.DataSource) (tlsSettings, error) {
tlsMode := strings.TrimSpace(strings.ToLower(datasource.JsonData.Get("sslmode").MustString("verify-full")))
isTLSDisabled := tlsMode == "disable"
func (m *tlsManager) getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error) {
tlsconfig := tlsSettings{
Mode: dsInfo.JsonData.Mode,
}
settings := tlsSettings{}
settings.Mode = tlsMode
isTLSDisabled := (tlsconfig.Mode == "disable")
if isTLSDisabled {
m.logger.Debug("Postgres TLS/SSL is disabled")
return settings, nil
return tlsconfig, nil
}
m.logger.Debug("Postgres TLS/SSL is enabled", "tlsMode", tlsMode)
m.logger.Debug("Postgres TLS/SSL is enabled", "tlsMode", tlsconfig.Mode)
settings.ConfigurationMethod = strings.TrimSpace(
strings.ToLower(datasource.JsonData.Get("tlsConfigurationMethod").MustString("file-path")))
tlsconfig.ConfigurationMethod = dsInfo.JsonData.ConfigurationMethod
tlsconfig.RootCertFile = dsInfo.JsonData.RootCertFile
tlsconfig.CertFile = dsInfo.JsonData.CertFile
tlsconfig.CertKeyFile = dsInfo.JsonData.CertKeyFile
if settings.ConfigurationMethod == "file-content" {
if err := m.writeCertFiles(datasource, &settings); err != nil {
return settings, err
if tlsconfig.ConfigurationMethod == "file-content" {
if err := m.writeCertFiles(dsInfo, &tlsconfig); err != nil {
return tlsconfig, err
}
} else {
settings.RootCertFile = datasource.JsonData.Get("sslRootCertFile").MustString("")
settings.CertFile = datasource.JsonData.Get("sslCertFile").MustString("")
settings.CertKeyFile = datasource.JsonData.Get("sslKeyFile").MustString("")
if err := validateCertFunc(settings.RootCertFile, settings.CertFile, settings.CertKeyFile); err != nil {
return settings, err
if err := validateCertFunc(tlsconfig.RootCertFile, tlsconfig.CertFile, tlsconfig.CertKeyFile); err != nil {
return tlsconfig, err
}
}
return settings, nil
return tlsconfig, nil
}
type certFileType int
const (
rootCert = iota
clientCert
clientKey
)
func (t certFileType) String() string {
switch t {
case rootCert:
@@ -118,8 +118,7 @@ func getFileName(dataDir string, fileType certFileType) string {
}
// writeCertFile writes a certificate file.
func writeCertFile(
ds *models.DataSource, logger log.Logger, fileContent string, generatedFilePath string) error {
func writeCertFile(logger log.Logger, fileContent string, generatedFilePath string) error {
fileContent = strings.TrimSpace(fileContent)
if fileContent != "" {
logger.Debug("Writing cert file", "path", generatedFilePath)
@@ -146,30 +145,28 @@ func writeCertFile(
return nil
}
func (m *tlsManager) writeCertFiles(ds *models.DataSource, settings *tlsSettings) error {
func (m *tlsManager) writeCertFiles(dsInfo sqleng.DataSourceInfo, tlsconfig *tlsSettings) error {
m.logger.Debug("Writing TLS certificate files to disk")
decrypted := ds.DecryptedValues()
tlsRootCert := decrypted["tlsCACert"]
tlsClientCert := decrypted["tlsClientCert"]
tlsClientKey := decrypted["tlsClientKey"]
tlsRootCert := dsInfo.DecryptedSecureJSONData["tlsCACert"]
tlsClientCert := dsInfo.DecryptedSecureJSONData["tlsClientCert"]
tlsClientKey := dsInfo.DecryptedSecureJSONData["tlsClientKey"]
if tlsRootCert == "" && tlsClientCert == "" && tlsClientKey == "" {
m.logger.Debug("No TLS/SSL certificates provided")
}
// Calculate all files path
workDir := filepath.Join(m.dataPath, "tls", ds.Uid+"generatedTLSCerts")
settings.RootCertFile = getFileName(workDir, rootCert)
settings.CertFile = getFileName(workDir, clientCert)
settings.CertKeyFile = getFileName(workDir, clientKey)
workDir := filepath.Join(m.dataPath, "tls", dsInfo.UID+"generatedTLSCerts")
tlsconfig.RootCertFile = getFileName(workDir, rootCert)
tlsconfig.CertFile = getFileName(workDir, clientCert)
tlsconfig.CertKeyFile = getFileName(workDir, clientKey)
// Find datasource in the cache, if found, skip writing files
cacheKey := strconv.Itoa(int(ds.Id))
cacheKey := strconv.Itoa(int(dsInfo.ID))
m.dsCacheInstance.locker.RLock(cacheKey)
item, ok := m.dsCacheInstance.cache.Load(cacheKey)
m.dsCacheInstance.locker.RUnlock(cacheKey)
if ok {
if item.(int) == ds.Version {
if !item.(time.Time).Before(dsInfo.Updated) {
return nil
}
}
@@ -179,7 +176,7 @@ func (m *tlsManager) writeCertFiles(ds *models.DataSource, settings *tlsSettings
item, ok = m.dsCacheInstance.cache.Load(cacheKey)
if ok {
if item.(int) == ds.Version {
if !item.(time.Time).Before(dsInfo.Updated) {
return nil
}
}
@@ -195,18 +192,18 @@ func (m *tlsManager) writeCertFiles(ds *models.DataSource, settings *tlsSettings
}
}
if err = writeCertFileFunc(ds, m.logger, tlsRootCert, settings.RootCertFile); err != nil {
if err = writeCertFileFunc(m.logger, tlsRootCert, tlsconfig.RootCertFile); err != nil {
return err
}
if err = writeCertFileFunc(ds, m.logger, tlsClientCert, settings.CertFile); err != nil {
if err = writeCertFileFunc(m.logger, tlsClientCert, tlsconfig.CertFile); err != nil {
return err
}
if err = writeCertFileFunc(ds, m.logger, tlsClientKey, settings.CertKeyFile); err != nil {
if err = writeCertFileFunc(m.logger, tlsClientKey, tlsconfig.CertKeyFile); err != nil {
return err
}
// Update datasource cache
m.dsCacheInstance.cache.Store(cacheKey, ds.Version)
m.dsCacheInstance.cache.Store(cacheKey, dsInfo.Updated)
return nil
}
+71 -69
View File
@@ -7,12 +7,11 @@ import (
"strings"
"sync"
"testing"
"time"
"github.com/grafana/grafana/pkg/components/securejsondata"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/tsdb/sqleng"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -30,16 +29,17 @@ func TestDataSourceCacheManager(t *testing.T) {
dsCacheInstance: datasourceCacheManager{locker: newLocker()},
dataPath: cfg.DataPath,
}
jsonData := simplejson.NewFromAny(map[string]interface{}{
"sslmode": "verify-full",
"tlsConfigurationMethod": "file-content",
})
secureJSONData := securejsondata.GetEncryptedJsonData(map[string]string{
jsonData := sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-content",
}
secureJSONData := map[string]string{
"tlsClientCert": "I am client certification",
"tlsClientKey": "I am client key",
"tlsCACert": "I am CA certification",
})
}
updateTime := time.Now().Add(-5 * time.Minute)
mockValidateCertFilePaths()
t.Cleanup(resetValidateCertFilePaths)
@@ -49,13 +49,13 @@ func TestDataSourceCacheManager(t *testing.T) {
wg.Add(10)
for id := int64(1); id <= 10; id++ {
go func(id int64) {
ds := &models.DataSource{
Id: id,
Version: 1,
Database: "database",
JsonData: jsonData,
SecureJsonData: secureJSONData,
Uid: "testData",
ds := sqleng.DataSourceInfo{
ID: id,
Updated: updateTime,
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(ds, &s)
@@ -67,9 +67,9 @@ func TestDataSourceCacheManager(t *testing.T) {
t.Run("check cache creation is succeed", func(t *testing.T) {
for id := int64(1); id <= 10; id++ {
version, ok := mng.dsCacheInstance.cache.Load(strconv.Itoa(int(id)))
updated, ok := mng.dsCacheInstance.cache.Load(strconv.Itoa(int(id)))
require.True(t, ok)
require.Equal(t, int(1), version)
require.Equal(t, updateTime, updated)
}
})
})
@@ -82,13 +82,13 @@ func TestDataSourceCacheManager(t *testing.T) {
wg1.Add(5)
for id := int64(1); id <= 5; id++ {
go func(id int64) {
ds := &models.DataSource{
Id: 1,
Version: 2,
Database: "database",
JsonData: jsonData,
SecureJsonData: secureJSONData,
Uid: "testData",
ds := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime,
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(ds, &s)
@@ -97,25 +97,25 @@ func TestDataSourceCacheManager(t *testing.T) {
}(id)
}
wg1.Wait()
assert.Equal(t, writeCertFileCallNum, 3)
assert.Equal(t, writeCertFileCallNum, 0)
})
t.Run("cache is updated with the last datasource version", func(t *testing.T) {
dsV2 := &models.DataSource{
Id: 1,
Version: 2,
Database: "database",
JsonData: jsonData,
SecureJsonData: secureJSONData,
Uid: "testData",
dsV2 := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime.Add(time.Minute),
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
dsV3 := &models.DataSource{
Id: 1,
Version: 3,
Database: "database",
JsonData: jsonData,
SecureJsonData: secureJSONData,
Uid: "testData",
dsV3 := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime.Add(2 * time.Minute),
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(dsV2, &s)
@@ -124,7 +124,7 @@ func TestDataSourceCacheManager(t *testing.T) {
require.NoError(t, err)
version, ok := mng.dsCacheInstance.cache.Load("1")
require.True(t, ok)
require.Equal(t, int(3), version)
require.Equal(t, updateTime.Add(2*time.Minute), version)
})
})
}
@@ -173,36 +173,39 @@ func TestGetTLSSettings(t *testing.T) {
mockValidateCertFilePaths()
t.Cleanup(resetValidateCertFilePaths)
updatedTime := time.Now()
testCases := []struct {
desc string
expErr string
jsonData map[string]interface{}
jsonData sqleng.JsonData
secureJSONData map[string]string
uid string
tlsSettings tlsSettings
version int
updated time.Time
}{
{
desc: "Custom TLS authentication disabled",
version: 1,
jsonData: map[string]interface{}{
"sslmode": "disable",
"sslRootCertFile": "i/am/coding/ca.crt",
"sslCertFile": "i/am/coding/client.crt",
"sslKeyFile": "i/am/coding/client.key",
"tlsConfigurationMethod": "file-path",
updated: updatedTime,
jsonData: sqleng.JsonData{
Mode: "disable",
RootCertFile: "i/am/coding/ca.crt",
CertFile: "i/am/coding/client.crt",
CertKeyFile: "i/am/coding/client.key",
ConfigurationMethod: "file-path",
},
tlsSettings: tlsSettings{Mode: "disable"},
},
{
desc: "Custom TLS authentication with file path",
version: 2,
jsonData: map[string]interface{}{
"sslmode": "verify-full",
"sslRootCertFile": "i/am/coding/ca.crt",
"sslCertFile": "i/am/coding/client.crt",
"sslKeyFile": "i/am/coding/client.key",
"tlsConfigurationMethod": "file-path",
updated: updatedTime.Add(time.Minute),
jsonData: sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-path",
RootCertFile: "i/am/coding/ca.crt",
CertFile: "i/am/coding/client.crt",
CertKeyFile: "i/am/coding/client.key",
},
tlsSettings: tlsSettings{
Mode: "verify-full",
@@ -214,11 +217,11 @@ func TestGetTLSSettings(t *testing.T) {
},
{
desc: "Custom TLS mode verify-full with certificate files content",
version: 3,
updated: updatedTime.Add(2 * time.Minute),
uid: "xxx",
jsonData: map[string]interface{}{
"sslmode": "verify-full",
"tlsConfigurationMethod": "file-content",
jsonData: sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-content",
},
secureJSONData: map[string]string{
"tlsCACert": "I am CA certification",
@@ -244,12 +247,11 @@ func TestGetTLSSettings(t *testing.T) {
dataPath: cfg.DataPath,
}
jsonData := simplejson.NewFromAny(tt.jsonData)
ds := &models.DataSource{
JsonData: jsonData,
SecureJsonData: securejsondata.GetEncryptedJsonData(tt.secureJSONData),
Uid: tt.uid,
Version: tt.version,
ds := sqleng.DataSourceInfo{
JsonData: tt.jsonData,
DecryptedSecureJSONData: tt.secureJSONData,
UID: tt.uid,
Updated: tt.updated,
}
settings, err = mng.getTLSSettings(ds)
@@ -278,7 +280,7 @@ func resetValidateCertFilePaths() {
func mockWriteCertFile() {
writeCertFileCallNum = 0
writeCertFileFunc = func(ds *models.DataSource, logger log.Logger, fileContent string, generatedFilePath string) error {
writeCertFileFunc = func(logger log.Logger, fileContent string, generatedFilePath string) error {
writeCertFileCallNum++
return nil
}
+5 -5
View File
@@ -20,7 +20,7 @@ import (
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/plugins/backendplugin"
"github.com/grafana/grafana/pkg/plugins/backendplugin/coreplugin"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"github.com/opentracing/opentracing-go"
"github.com/prometheus/client_golang/api"
apiv1 "github.com/prometheus/client_golang/api/prometheus/v1"
@@ -43,7 +43,7 @@ type DatasourceInfo struct {
type Service struct {
httpClientProvider httpclient.Provider
intervalCalculator tsdb.Calculator
intervalCalculator intervalv2.Calculator
im instancemgmt.InstanceManager
}
@@ -53,7 +53,7 @@ func ProvideService(httpClientProvider httpclient.Provider, backendPluginManager
s := &Service{
httpClientProvider: httpClientProvider,
intervalCalculator: tsdb.NewCalculator(),
intervalCalculator: intervalv2.NewCalculator(),
im: im,
}
@@ -247,7 +247,7 @@ func (s *Service) parseQuery(queries []backend.DataQuery, dsInfo *DatasourceInfo
end := queryModel.TimeRange.To
queryInterval := jsonModel.Get("interval").MustString("")
foundInterval, err := tsdb.GetIntervalFrom(dsInfo.TimeInterval, queryInterval, 0, 15*time.Second)
foundInterval, err := intervalv2.GetIntervalFrom(dsInfo.TimeInterval, queryInterval, 0, 15*time.Second)
hasQueryInterval := queryInterval != ""
// Only use stepMode if we have interval in query, otherwise use "min"
if hasQueryInterval {
@@ -261,7 +261,7 @@ func (s *Service) parseQuery(queries []backend.DataQuery, dsInfo *DatasourceInfo
return nil, err
}
calculatedInterval, err := s.intervalCalculator.Calculate(queries[0].TimeRange, foundInterval, tsdb.IntervalMode(intervalMode))
calculatedInterval, err := s.intervalCalculator.Calculate(queries[0].TimeRange, foundInterval, intervalv2.IntervalMode(intervalMode))
if err != nil {
return nil, err
}
+2 -2
View File
@@ -6,7 +6,7 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana/pkg/tsdb"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
p "github.com/prometheus/common/model"
"github.com/stretchr/testify/require"
)
@@ -15,7 +15,7 @@ var now = time.Now()
func TestPrometheus(t *testing.T) {
service := Service{
intervalCalculator: tsdb.NewCalculator(),
intervalCalculator: intervalv2.NewCalculator(),
}
t.Run("converting metric name", func(t *testing.T) {
+1 -7
View File
@@ -11,24 +11,18 @@ import (
"github.com/grafana/grafana/pkg/services/oauthtoken"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/tsdb/cloudmonitoring"
"github.com/grafana/grafana/pkg/tsdb/mssql"
"github.com/grafana/grafana/pkg/tsdb/mysql"
"github.com/grafana/grafana/pkg/tsdb/postgres"
_ "github.com/grafana/grafana/pkg/tsdb/postgres"
)
// NewService returns a new Service.
func NewService(
cfg *setting.Cfg, pluginManager plugins.Manager, backendPluginManager backendplugin.Manager,
oauthTokenService *oauthtoken.Service, httpClientProvider httpclient.Provider, cloudMonitoringService *cloudmonitoring.Service,
postgresService *postgres.PostgresService,
) *Service {
s := newService(cfg, pluginManager, backendPluginManager, oauthTokenService)
// register backend data sources using legacy plugin
// contracts/non-SDK contracts
s.registry["mssql"] = mssql.NewExecutor
s.registry["postgres"] = postgresService.NewExecutor
s.registry["mysql"] = mysql.New(httpClientProvider)
s.registry["stackdriver"] = cloudMonitoringService.NewExecutor
return s
+148 -102
View File
@@ -3,6 +3,7 @@ package sqleng
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"net"
@@ -15,11 +16,8 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/plugins"
"github.com/grafana/grafana/pkg/tsdb/interval"
"github.com/grafana/grafana/pkg/tsdb/intervalv2"
"xorm.io/core"
"xorm.io/xorm"
)
@@ -32,29 +30,28 @@ var ErrConnectionFailed = errors.New("failed to connect to server - please inspe
// SQLMacroEngine interpolates macros into sql. It takes in the Query to have access to query context and
// timeRange to be able to generate queries that use from and to.
type SQLMacroEngine interface {
Interpolate(query plugins.DataSubQuery, timeRange plugins.DataTimeRange, sql string) (string, error)
Interpolate(query *backend.DataQuery, timeRange backend.TimeRange, sql string) (string, error)
}
// SqlQueryResultTransformer transforms a query result row to RowValues with proper types.
type SqlQueryResultTransformer interface {
// TransformQueryError transforms a query error.
TransformQueryError(err error) error
GetConverterList() []sqlutil.StringConverter
}
type engineCacheType struct {
cache map[int64]*xorm.Engine
versions map[int64]int
cache map[int64]*xorm.Engine
updates map[int64]time.Time
sync.Mutex
}
var engineCache = engineCacheType{
cache: make(map[int64]*xorm.Engine),
versions: make(map[int64]int),
cache: make(map[int64]*xorm.Engine),
updates: make(map[int64]time.Time),
}
var sqlIntervalCalculator = interval.NewCalculator()
var sqlIntervalCalculator = intervalv2.NewCalculator()
// NewXormEngine is an xorm.Engine factory, that can be stubbed by tests.
//nolint:gocritic
@@ -62,24 +59,58 @@ var NewXormEngine = func(driverName string, connectionString string) (*xorm.Engi
return xorm.NewEngine(driverName, connectionString)
}
type dataPlugin struct {
type JsonData struct {
MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"`
ConnMaxLifetime int `json:"connMaxLifetime"`
Timescaledb bool `json:"timescaledb"`
Mode string `json:"sslmode"`
ConfigurationMethod string `json:"tlsConfigurationMethod"`
RootCertFile string `json:"sslRootCertFile"`
CertFile string `json:"sslCertFile"`
CertKeyFile string `json:"sslKeyFile"`
Timezone string `json:"timezone"`
Encrypt string `json:"encrypt"`
TimeInterval string `json:"timeInterval"`
}
type DataSourceInfo struct {
JsonData JsonData
URL string
User string
Database string
ID int64
Updated time.Time
UID string
DecryptedSecureJSONData map[string]string
}
type DataPluginConfiguration struct {
DriverName string
DSInfo DataSourceInfo
ConnectionString string
TimeColumnNames []string
MetricColumnTypes []string
}
type DataSourceHandler struct {
macroEngine SQLMacroEngine
queryResultTransformer SqlQueryResultTransformer
engine *xorm.Engine
timeColumnNames []string
metricColumnTypes []string
log log.Logger
dsInfo DataSourceInfo
}
type QueryJson struct {
RawSql string `json:"rawSql"`
Fill bool `json:"fill"`
FillInterval float64 `json:"fillInterval"`
FillMode string `json:"fillMode"`
FillValue float64 `json:"fillValue"`
Format string `json:"format"`
}
type DataPluginConfiguration struct {
DriverName string
Datasource *models.DataSource
ConnectionString string
TimeColumnNames []string
MetricColumnTypes []string
}
func (e *dataPlugin) transformQueryError(err error) error {
func (e *DataSourceHandler) transformQueryError(err error) error {
// OpError is the error type usually returned by functions in the net
// package. It describes the operation, network type, and address of
// an error. We log this error rather than return it to the client
@@ -93,32 +124,31 @@ func (e *dataPlugin) transformQueryError(err error) error {
return e.queryResultTransformer.TransformQueryError(err)
}
// NewDataPlugin returns a new plugins.DataPlugin
//nolint: staticcheck // plugins.DataPlugin deprecated
func NewDataPlugin(config DataPluginConfiguration, queryResultTransformer SqlQueryResultTransformer,
macroEngine SQLMacroEngine, log log.Logger) (plugins.DataPlugin, error) {
plugin := dataPlugin{
func NewQueryDataHandler(config DataPluginConfiguration, queryResultTransformer SqlQueryResultTransformer,
macroEngine SQLMacroEngine, log log.Logger) (*DataSourceHandler, error) {
queryDataHandler := DataSourceHandler{
queryResultTransformer: queryResultTransformer,
macroEngine: macroEngine,
timeColumnNames: []string{"time"},
log: log,
dsInfo: config.DSInfo,
}
if len(config.TimeColumnNames) > 0 {
plugin.timeColumnNames = config.TimeColumnNames
queryDataHandler.timeColumnNames = config.TimeColumnNames
}
if len(config.MetricColumnTypes) > 0 {
plugin.metricColumnTypes = config.MetricColumnTypes
queryDataHandler.metricColumnTypes = config.MetricColumnTypes
}
engineCache.Lock()
defer engineCache.Unlock()
if engine, present := engineCache.cache[config.Datasource.Id]; present {
if version := engineCache.versions[config.Datasource.Id]; version == config.Datasource.Version {
plugin.engine = engine
return &plugin, nil
if engine, present := engineCache.cache[config.DSInfo.ID]; present {
if updateTime := engineCache.updates[config.DSInfo.ID]; updateTime.Before(config.DSInfo.Updated) {
queryDataHandler.engine = engine
return &queryDataHandler, nil
}
}
@@ -127,104 +157,104 @@ func NewDataPlugin(config DataPluginConfiguration, queryResultTransformer SqlQue
return nil, err
}
maxOpenConns := config.Datasource.JsonData.Get("maxOpenConns").MustInt(0)
engine.SetMaxOpenConns(maxOpenConns)
maxIdleConns := config.Datasource.JsonData.Get("maxIdleConns").MustInt(2)
engine.SetMaxIdleConns(maxIdleConns)
connMaxLifetime := config.Datasource.JsonData.Get("connMaxLifetime").MustInt(14400)
engine.SetConnMaxLifetime(time.Duration(connMaxLifetime) * time.Second)
engine.SetMaxOpenConns(config.DSInfo.JsonData.MaxOpenConns)
engine.SetMaxIdleConns(config.DSInfo.JsonData.MaxIdleConns)
engine.SetConnMaxLifetime(time.Duration(config.DSInfo.JsonData.ConnMaxLifetime) * time.Second)
engineCache.versions[config.Datasource.Id] = config.Datasource.Version
engineCache.cache[config.Datasource.Id] = engine
plugin.engine = engine
return &plugin, nil
engineCache.updates[config.DSInfo.ID] = config.DSInfo.Updated
engineCache.cache[config.DSInfo.ID] = engine
queryDataHandler.engine = engine
return &queryDataHandler, nil
}
const rowLimit = 1000000
// DataQuery queries for data.
//nolint: staticcheck // plugins.DataPlugin deprecated
func (e *dataPlugin) DataQuery(ctx context.Context, dsInfo *models.DataSource,
queryContext plugins.DataQuery) (plugins.DataResponse, error) {
ch := make(chan plugins.DataQueryResult, len(queryContext.Queries))
type DBDataResponse struct {
dataResponse backend.DataResponse
refID string
}
func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
result := backend.NewQueryDataResponse()
ch := make(chan DBDataResponse, len(req.Queries))
var wg sync.WaitGroup
// Execute each query in a goroutine and wait for them to finish afterwards
for _, query := range queryContext.Queries {
if query.Model.Get("rawSql").MustString() == "" {
for _, query := range req.Queries {
queryjson := QueryJson{
Fill: false,
Format: "time_series",
}
err := json.Unmarshal(query.JSON, &queryjson)
if err != nil {
return nil, fmt.Errorf("error unmarshal query json: %w", err)
}
if queryjson.RawSql == "" {
continue
}
wg.Add(1)
go e.executeQuery(query, &wg, queryContext, ch)
go e.executeQuery(query, &wg, ctx, ch, queryjson)
}
wg.Wait()
// Read results from channels
close(ch)
result := plugins.DataResponse{
Results: make(map[string]plugins.DataQueryResult),
}
result.Responses = make(map[string]backend.DataResponse)
for queryResult := range ch {
result.Results[queryResult.RefID] = queryResult
result.Responses[queryResult.refID] = queryResult.dataResponse
}
return result, nil
}
//nolint: staticcheck,gocyclo // plugins.DataQueryResult deprecated
func (e *dataPlugin) executeQuery(query plugins.DataSubQuery, wg *sync.WaitGroup, queryContext plugins.DataQuery,
ch chan plugins.DataQueryResult) {
func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitGroup, queryContext context.Context,
ch chan DBDataResponse, queryJson QueryJson) {
defer wg.Done()
queryResult := plugins.DataQueryResult{
Meta: simplejson.New(),
RefID: query.RefID,
queryResult := DBDataResponse{
dataResponse: backend.DataResponse{},
refID: query.RefID,
}
defer func() {
if r := recover(); r != nil {
e.log.Error("executeQuery panic", "error", r, "stack", log.Stack(1))
if theErr, ok := r.(error); ok {
queryResult.Error = theErr
queryResult.dataResponse.Error = theErr
} else if theErrString, ok := r.(string); ok {
queryResult.Error = fmt.Errorf(theErrString)
queryResult.dataResponse.Error = fmt.Errorf(theErrString)
} else {
queryResult.Error = fmt.Errorf("unexpected error, see the server log for details")
queryResult.dataResponse.Error = fmt.Errorf("unexpected error, see the server log for details")
}
ch <- queryResult
}
}()
rawSQL := query.Model.Get("rawSql").MustString()
if rawSQL == "" {
if queryJson.RawSql == "" {
panic("Query model property rawSql should not be empty at this point")
}
var timeRange plugins.DataTimeRange
if queryContext.TimeRange != nil {
timeRange = *queryContext.TimeRange
}
timeRange := query.TimeRange
errAppendDebug := func(frameErr string, err error, query string) {
var emptyFrame data.Frame
emptyFrame.SetMeta(&data.FrameMeta{
ExecutedQueryString: query,
})
queryResult.Error = fmt.Errorf("%s: %w", frameErr, err)
queryResult.Dataframes = plugins.NewDecodedDataFrames(data.Frames{&emptyFrame})
queryResult.dataResponse.Error = fmt.Errorf("%s: %w", frameErr, err)
queryResult.dataResponse.Frames = data.Frames{&emptyFrame}
ch <- queryResult
}
// global substitutions
interpolatedQuery, err := Interpolate(query, timeRange, rawSQL)
interpolatedQuery, err := Interpolate(query, timeRange, e.dsInfo.JsonData.TimeInterval, queryJson.RawSql)
if err != nil {
errAppendDebug("interpolation failed", e.transformQueryError(err), interpolatedQuery)
return
}
// data source specific substitutions
interpolatedQuery, err = e.macroEngine.Interpolate(query, timeRange, interpolatedQuery)
interpolatedQuery, err = e.macroEngine.Interpolate(&query, timeRange, interpolatedQuery)
if err != nil {
errAppendDebug("interpolation failed", e.transformQueryError(err), interpolatedQuery)
return
@@ -265,7 +295,7 @@ func (e *dataPlugin) executeQuery(query plugins.DataSubQuery, wg *sync.WaitGroup
// If no rows were returned, no point checking anything else.
if frame.Rows() == 0 {
queryResult.Dataframes = plugins.NewDecodedDataFrames(data.Frames{frame})
queryResult.dataResponse.Frames = data.Frames{frame}
ch <- queryResult
return
}
@@ -338,13 +368,13 @@ func (e *dataPlugin) executeQuery(query plugins.DataSubQuery, wg *sync.WaitGroup
}
}
queryResult.Dataframes = plugins.NewDecodedDataFrames(data.Frames{frame})
queryResult.dataResponse.Frames = data.Frames{frame}
ch <- queryResult
}
// Interpolate provides global macros/substitutions for all sql datasources.
var Interpolate = func(query plugins.DataSubQuery, timeRange plugins.DataTimeRange, sql string) (string, error) {
minInterval, err := interval.GetIntervalFrom(query.DataSource, query.Model, time.Second*60)
var Interpolate = func(query backend.DataQuery, timeRange backend.TimeRange, timeInterval string, sql string) (string, error) {
minInterval, err := intervalv2.GetIntervalFrom(timeInterval, query.Interval.String(), query.Interval.Milliseconds(), time.Second*60)
if err != nil {
return "", err
}
@@ -355,14 +385,14 @@ var Interpolate = func(query plugins.DataSubQuery, timeRange plugins.DataTimeRan
sql = strings.ReplaceAll(sql, "$__interval_ms", strconv.FormatInt(interval.Milliseconds(), 10))
sql = strings.ReplaceAll(sql, "$__interval", interval.Text)
sql = strings.ReplaceAll(sql, "$__unixEpochFrom()", fmt.Sprintf("%d", timeRange.GetFromAsSecondsEpoch()))
sql = strings.ReplaceAll(sql, "$__unixEpochTo()", fmt.Sprintf("%d", timeRange.GetToAsSecondsEpoch()))
sql = strings.ReplaceAll(sql, "$__unixEpochFrom()", fmt.Sprintf("%d", timeRange.From.UTC().Unix()))
sql = strings.ReplaceAll(sql, "$__unixEpochTo()", fmt.Sprintf("%d", timeRange.To.UTC().Unix()))
return sql, nil
}
//nolint: staticcheck // plugins.DataPlugin deprecated
func (e *dataPlugin) newProcessCfg(query plugins.DataSubQuery, queryContext plugins.DataQuery,
func (e *DataSourceHandler) newProcessCfg(query backend.DataQuery, queryContext context.Context,
rows *core.Rows, interpolatedQuery string) (*dataQueryModel, error) {
columnNames, err := rows.Columns()
if err != nil {
@@ -383,35 +413,38 @@ func (e *dataPlugin) newProcessCfg(query plugins.DataSubQuery, queryContext plug
queryContext: queryContext,
}
if query.Model.Get("fill").MustBool(false) {
queryJson := QueryJson{}
err = json.Unmarshal(query.JSON, &queryJson)
if err != nil {
return nil, err
}
if queryJson.Fill {
qm.FillMissing = &data.FillMissing{}
qm.Interval = time.Duration(query.Model.Get("fillInterval").MustFloat64() * float64(time.Second))
switch strings.ToLower(query.Model.Get("fillMode").MustString()) {
qm.Interval = time.Duration(queryJson.FillInterval * float64(time.Second))
switch strings.ToLower(queryJson.FillMode) {
case "null":
qm.FillMissing.Mode = data.FillModeNull
case "previous":
qm.FillMissing.Mode = data.FillModePrevious
case "value":
qm.FillMissing.Mode = data.FillModeValue
qm.FillMissing.Value = query.Model.Get("fillValue").MustFloat64()
qm.FillMissing.Value = queryJson.FillValue
default:
}
}
//nolint: staticcheck // plugins.DataPlugin deprecated
if queryContext.TimeRange != nil {
qm.TimeRange.From = queryContext.TimeRange.GetFromAsTimeUTC()
qm.TimeRange.To = queryContext.TimeRange.GetToAsTimeUTC()
}
qm.TimeRange.From = query.TimeRange.From.UTC()
qm.TimeRange.To = query.TimeRange.To.UTC()
format := query.Model.Get("format").MustString("time_series")
switch format {
switch queryJson.Format {
case "time_series":
qm.Format = dataQueryFormatSeries
case "table":
qm.Format = dataQueryFormatTable
default:
panic(fmt.Sprintf("Unrecognized query model format: %q", format))
panic(fmt.Sprintf("Unrecognized query model format: %q", queryJson.Format))
}
for i, col := range qm.columnNames {
@@ -462,7 +495,7 @@ type dataQueryModel struct {
metricIndex int
rows *core.Rows
metricPrefix bool
queryContext plugins.DataQuery
queryContext context.Context
}
func convertInt64ToFloat64(origin *data.Field, newField *data.Field) {
@@ -902,23 +935,36 @@ func convertSQLValueColumnToFloat(frame *data.Frame, Index int) (*data.Frame, er
return frame, nil
}
func SetupFillmode(query plugins.DataSubQuery, interval time.Duration, fillmode string) error {
query.Model.Set("fill", true)
query.Model.Set("fillInterval", interval.Seconds())
func SetupFillmode(query *backend.DataQuery, interval time.Duration, fillmode string) error {
rawQueryProp := make(map[string]interface{})
queryBytes, err := query.JSON.MarshalJSON()
if err != nil {
return err
}
err = json.Unmarshal(queryBytes, &rawQueryProp)
if err != nil {
return err
}
rawQueryProp["fill"] = true
rawQueryProp["fillInterval"] = interval.Seconds()
switch fillmode {
case "NULL":
query.Model.Set("fillMode", "null")
rawQueryProp["fillMode"] = "null"
case "previous":
query.Model.Set("fillMode", "previous")
rawQueryProp["fillMode"] = "previous"
default:
query.Model.Set("fillMode", "value")
rawQueryProp["fillMode"] = "value"
floatVal, err := strconv.ParseFloat(fillmode, 64)
if err != nil {
return fmt.Errorf("error parsing fill value %v", fillmode)
}
query.Model.Set("fillValue", floatVal)
rawQueryProp["fillValue"] = floatVal
}
query.JSON, err = json.Marshal(rawQueryProp)
if err != nil {
return err
}
return nil
}
+9 -10
View File
@@ -7,11 +7,10 @@ import (
"testing"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/components/simplejson"
"github.com/grafana/grafana/pkg/infra/log"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/plugins"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -25,35 +24,35 @@ func TestSQLEngine(t *testing.T) {
t.Run("Given a time range between 2018-04-12 00:00 and 2018-04-12 00:05", func(t *testing.T) {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := plugins.DataTimeRange{From: "5m", To: "now", Now: to}
query := plugins.DataSubQuery{DataSource: &models.DataSource{}, Model: simplejson.New()}
timeRange := backend.TimeRange{From: from, To: to}
query := backend.DataQuery{JSON: []byte("{}")}
t.Run("interpolate $__interval", func(t *testing.T) {
sql, err := Interpolate(query, timeRange, "select $__interval ")
sql, err := Interpolate(query, timeRange, "", "select $__interval ")
require.NoError(t, err)
require.Equal(t, "select 1m ", sql)
})
t.Run("interpolate $__interval in $__timeGroup", func(t *testing.T) {
sql, err := Interpolate(query, timeRange, "select $__timeGroupAlias(time,$__interval)")
sql, err := Interpolate(query, timeRange, "", "select $__timeGroupAlias(time,$__interval)")
require.NoError(t, err)
require.Equal(t, "select $__timeGroupAlias(time,1m)", sql)
})
t.Run("interpolate $__interval_ms", func(t *testing.T) {
sql, err := Interpolate(query, timeRange, "select $__interval_ms ")
sql, err := Interpolate(query, timeRange, "", "select $__interval_ms ")
require.NoError(t, err)
require.Equal(t, "select 60000 ", sql)
})
t.Run("interpolate __unixEpochFrom function", func(t *testing.T) {
sql, err := Interpolate(query, timeRange, "select $__unixEpochFrom()")
sql, err := Interpolate(query, timeRange, "", "select $__unixEpochFrom()")
require.NoError(t, err)
require.Equal(t, fmt.Sprintf("select %d", from.Unix()), sql)
})
t.Run("interpolate __unixEpochTo function", func(t *testing.T) {
sql, err := Interpolate(query, timeRange, "select $__unixEpochTo()")
sql, err := Interpolate(query, timeRange, "", "select $__unixEpochTo()")
require.NoError(t, err)
require.Equal(t, fmt.Sprintf("select %d", to.Unix()), sql)
})
@@ -406,7 +405,7 @@ func TestSQLEngine(t *testing.T) {
for _, tc := range tests {
transformer := &testQueryResultTransformer{}
dp := dataPlugin{
dp := DataSourceHandler{
log: log.New("test"),
queryResultTransformer: transformer,
}
+1 -2
View File
@@ -1,4 +1,3 @@
# Canvas panel - Native Plugin
# Canvas panel - Native Plugin
The Canvas Panel is **included** with Grafana.