Compare commits

..

4 Commits

Author SHA1 Message Date
Will Assis 861af005e0 unified-storage: fix listModifiedSince in storage_backend.go and enable its tests for both badger and sqlkv 2026-01-14 19:24:15 -03:00
Will Assis 14a05137e1 fmt 2026-01-14 15:43:30 -03:00
Will Assis cfe86378a1 dont run tests on sqlite (yet) 2026-01-14 15:41:51 -03:00
Will Assis f7d7e09626 unified-storage: sqlkv enable more tests 2026-01-14 15:25:42 -03:00
32 changed files with 246 additions and 988 deletions
@@ -270,17 +270,7 @@ Click **View in CloudWatch console** to interactively view, search, and analyze
### Query Log groups with OpenSearch SQL
When querying log groups with OpenSearch SQL, you can use the `$__logGroups` macro to automatically reference log groups selected in the query editor's log group selector. This is the recommended approach as it allows you to manage log groups through the UI.
```sql
SELECT window.start, COUNT(*) AS exceptionCount
FROM $__logGroups
WHERE `@message` LIKE '%Exception%'
```
The `$__logGroups` macro expands to the proper `logGroups(logGroupIdentifier: [...])` syntax with the log groups you've selected in the UI.
Alternatively, you can manually specify a single log group directly in the `FROM` clause:
When querying log groups with OpenSearch SQL, you **must** explicitly state the log group identifier or ARN in the `FROM` clause:
```sql
SELECT window.start, COUNT(*) AS exceptionCount
@@ -288,7 +278,7 @@ FROM `log_group`
WHERE `@message` LIKE '%Exception%'
```
When querying multiple log groups you **must** use the `logGroups(logGroupIdentifier: [...])` syntax:
or, when querying multiple log groups:
```sql
SELECT window.start, COUNT(*) AS exceptionCount
@@ -296,8 +286,6 @@ FROM `logGroups( logGroupIdentifier: ['LogGroup1', 'LogGroup2'])`
WHERE `@message` LIKE '%Exception%'
```
To reference log groups in a monitoring account, use ARNs instead of LogGroup names.
You can also write queries returning time series data by using the [`stats` command](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CWL_Insights-Visualizing-Log-Data.html).
When making `stats` queries in [Explore](ref:explore), ensure you are in Metrics Explore mode.
@@ -4,8 +4,7 @@ comments: |
This file is used in the following visualizations: candlestick, heatmap, state timeline, status history, time series.
---
You can pan the panel time range left and right, and zoom it and in and out.
This, in turn, changes the dashboard time range.
You can zoom the panel time range in and out, which in turn, changes the dashboard time range.
**Zoom in** - Click and drag on the panel to zoom in on a particular time range.
@@ -17,9 +16,4 @@ For example, if the original time range is from 9:00 to 9:59, the time range cha
- Next range: 8:30 - 10:29
- Next range: 7:30 - 11:29
**Pan** - Click and drag the x-axis area of the panel to pan the time range.
The time range shifts by the distance you drag.
For example, if the original time range is from 9:00 to 9:59 and you drag 30 minutes to the right, the time range changes to 9:30 to 10:29.
For screen recordings showing these interactions, refer to the [Panel overview documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/panel-overview/#pan-and-zoom-panel-time-range).
For screen recordings showing these interactions, refer to the [Panel overview documentation](https://grafana.com/docs/grafana/<GRAFANA_VERSION>/visualizations/panels-visualizations/panel-overview/#zoom-panel-time-range).
@@ -317,16 +317,13 @@ Click the **Copy time range to clipboard** icon to copy the current time range t
You can also copy and paste a time range using the keyboard shortcuts `t+c` and `t+v` respectively.
#### Zoom out
#### Zoom out (Cmd+Z or Ctrl+Z)
- Click the **Zoom out** icon to view a larger time range in the dashboard or panel visualizations
- Double click on the panel graph area (time series family visualizations only)
- Type the `t-` keyboard shortcut
Click the **Zoom out** icon to view a larger time range in the dashboard or panel visualization.
#### Zoom in
#### Zoom in (only applicable to graph visualizations)
- Click and drag horizontally in the panel graph area to select a time range (time series family visualizations only)
- Type the `t+` keyboard shortcut
Click and drag to select the time range in the visualization that you want to view.
#### Refresh dashboard
@@ -175,10 +175,9 @@ By hovering over a panel with the mouse you can use some shortcuts that will tar
- `pl`: Hide or show legend
- `pr`: Remove Panel
## Pan and zoom panel time range
## Zoom panel time range
You can pan the panel time range left and right, and zoom it and in and out.
This, in turn, changes the dashboard time range.
You can zoom the panel time range in and out, which in turn, changes the dashboard time range.
This feature is supported for the following visualizations:
@@ -192,7 +191,7 @@ This feature is supported for the following visualizations:
Click and drag on the panel to zoom in on a particular time range.
The following screen recordings show this interaction in the time series and candlestick visualizations:
The following screen recordings show this interaction in the time series and x visualizations:
Time series
@@ -212,7 +211,7 @@ For example, if the original time range is from 9:00 to 9:59, the time range cha
- Next range: 8:30 - 10:29
- Next range: 7:30 - 11:29
The following screen recordings demonstrate the preceding example in the time series and heatmap visualizations:
The following screen recordings demonstrate the preceding example in the time series and x visualizations:
Time series
@@ -222,19 +221,6 @@ Heatmap
{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-heatmap-panel-time-zoom-out-mouse.mp4" >}}
### Pan
Click and drag the x-axis area of the panel to pan the time range.
The time range shifts by the distance you drag.
For example, if the original time range is from 9:00 to 9:59 and you drag 30 minutes to the right, the time range changes to 9:30 to 10:29.
The following screen recordings show this interaction in the time series visualization:
Time series
{{< video-embed src="/media/docs/grafana/panels-visualizations/recording-ts-time-pan-mouse.mp4" >}}
## Add a panel
To add a panel in a new dashboard click **+ Add visualization** in the middle of the dashboard:
@@ -92,9 +92,9 @@ The data is converted as follows:
{{< figure src="/media/docs/grafana/panels-visualizations/screenshot-candles-volume-v11.6.png" max-width="750px" alt="A candlestick visualization showing the price movements of specific asset." >}}
## Pan and zoom panel time range
## Zoom panel time range
{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{{< docs/shared lookup="visualizations/panel-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
## Configuration options
@@ -79,9 +79,9 @@ The data is converted as follows:
{{< figure src="/static/img/docs/heatmap-panel/heatmap.png" max-width="1025px" alt="A heatmap visualization showing the random walk distribution over time" >}}
## Pan and zoom panel time range
## Zoom panel time range
{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{{< docs/shared lookup="visualizations/panel-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
## Configuration options
@@ -93,9 +93,9 @@ You can also create a state timeline visualization using time series data. To do
![State timeline with time series](/media/docs/grafana/panels-visualizations/screenshot-state-timeline-time-series-v11.4.png)
## Pan and zoom panel time range
## Zoom panel time range
{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{{< docs/shared lookup="visualizations/panel-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
## Configuration options
@@ -85,9 +85,9 @@ The data is converted as follows:
{{< figure src="/static/img/docs/status-history-panel/status_history.png" max-width="1025px" alt="A status history panel with two time columns showing the status of two servers" >}}
## Pan and zoom panel time range
## Zoom panel time range
{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{{< docs/shared lookup="visualizations/panel-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
## Configuration options
@@ -167,9 +167,9 @@ The following example shows three series: Min, Max, and Value. The Min and Max s
{{< docs/shared lookup="visualizations/multiple-y-axes.md" source="grafana" version="<GRAFANA_VERSION>" leveloffset="+2" >}}
## Pan and zoom panel time range
## Zoom panel time range
{{< docs/shared lookup="visualizations/panel-pan-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
{{< docs/shared lookup="visualizations/panel-zoom.md" source="grafana" version="<GRAFANA_VERSION>" >}}
## Configuration options
+18 -24
View File
@@ -76,27 +76,21 @@ func (hs *HTTPServer) CreateDashboardSnapshot(c *contextmodel.ReqContext) {
return
}
cfg := snapshot.SnapshotSharingOptions{
// Do not check permissions when the instance snapshot public mode is enabled
if !hs.Cfg.SnapshotPublicMode {
evaluator := ac.EvalAll(ac.EvalPermission(dashboards.ActionSnapshotsCreate), ac.EvalPermission(dashboards.ActionDashboardsRead, dashboards.ScopeDashboardsProvider.GetResourceScopeUID(cmd.Dashboard.GetNestedString("uid"))))
if canSave, err := hs.AccessControl.Evaluate(c.Req.Context(), c.SignedInUser, evaluator); err != nil || !canSave {
c.JsonApiErr(http.StatusForbidden, "forbidden", err)
return
}
}
dashboardsnapshots.CreateDashboardSnapshot(c, snapshot.SnapshotSharingOptions{
SnapshotsEnabled: hs.Cfg.SnapshotEnabled,
ExternalEnabled: hs.Cfg.ExternalEnabled,
ExternalSnapshotName: hs.Cfg.ExternalSnapshotName,
ExternalSnapshotURL: hs.Cfg.ExternalSnapshotUrl,
}
if hs.Cfg.SnapshotPublicMode {
// Public mode: no user or dashboard validation needed
dashboardsnapshots.CreateDashboardSnapshotPublic(c, cfg, cmd, hs.dashboardsnapshotsService)
return
}
// Regular mode: check permissions
evaluator := ac.EvalAll(ac.EvalPermission(dashboards.ActionSnapshotsCreate), ac.EvalPermission(dashboards.ActionDashboardsRead, dashboards.ScopeDashboardsProvider.GetResourceScopeUID(cmd.Dashboard.GetNestedString("uid"))))
if canSave, err := hs.AccessControl.Evaluate(c.Req.Context(), c.SignedInUser, evaluator); err != nil || !canSave {
c.JsonApiErr(http.StatusForbidden, "forbidden", err)
return
}
dashboardsnapshots.CreateDashboardSnapshot(c, cfg, cmd, hs.dashboardsnapshotsService)
}, cmd, hs.dashboardsnapshotsService)
}
// GET /api/snapshots/:key
@@ -219,6 +213,13 @@ func (hs *HTTPServer) DeleteDashboardSnapshot(c *contextmodel.ReqContext) respon
return response.Error(http.StatusUnauthorized, "OrgID mismatch", nil)
}
if queryResult.External {
err := dashboardsnapshots.DeleteExternalDashboardSnapshot(queryResult.ExternalDeleteURL)
if err != nil {
return response.Error(http.StatusInternalServerError, "Failed to delete external dashboard", err)
}
}
// Dashboard can be empty (creation error or external snapshot). This means that the mustInt here returns a 0,
// which before RBAC would result in a dashboard which has no ACL. A dashboard without an ACL would fallback
// to the users org role, which for editors and admins would essentially always be allowed here. With RBAC,
@@ -238,13 +239,6 @@ func (hs *HTTPServer) DeleteDashboardSnapshot(c *contextmodel.ReqContext) respon
}
}
if queryResult.External {
err := dashboardsnapshots.DeleteExternalDashboardSnapshot(queryResult.ExternalDeleteURL)
if err != nil {
return response.Error(http.StatusInternalServerError, "Failed to delete external dashboard", err)
}
}
cmd := &dashboardsnapshots.DeleteDashboardSnapshotCommand{DeleteKey: queryResult.DeleteKey}
if err := hs.dashboardsnapshotsService.DeleteDashboardSnapshot(c.Req.Context(), cmd); err != nil {
+1 -12
View File
@@ -32,8 +32,6 @@ import (
var (
logger = glog.New("data-proxy-log")
client = newHTTPClient()
errPluginProxyRouteAccessDenied = errors.New("plugin proxy route access denied")
)
type DataSourceProxy struct {
@@ -310,21 +308,12 @@ func (proxy *DataSourceProxy) validateRequest() error {
if err != nil {
return err
}
// issues/116273: When we have an empty input route (or input that becomes relative to "."), we do not want it
// to be ".". This is because the `CleanRelativePath` function will never return "./" prefixes, and as such,
// the common prefix we need is an empty string.
if r1 == "." && proxy.proxyPath != "." {
r1 = ""
}
if r2 == "." && route.Path != "." {
r2 = ""
}
if !strings.HasPrefix(r1, r2) {
continue
}
if !proxy.hasAccessToRoute(route) {
return errPluginProxyRouteAccessDenied
return errors.New("plugin proxy route access denied")
}
proxy.matchedRoute = route
-88
View File
@@ -673,94 +673,6 @@ func TestIntegrationDataSourceProxy_routeRule(t *testing.T) {
runDatasourceAuthTest(t, secretsService, secretsStore, cfg, test)
}
})
t.Run("Regression of 116273: Fallback routes should apply fallback route roles", func(t *testing.T) {
for _, tc := range []struct {
InputPath string
ConfigurationPath string
ExpectError bool
}{
{
InputPath: "api/v2/leak-ur-secrets",
ConfigurationPath: "",
ExpectError: true,
},
{
InputPath: "",
ConfigurationPath: "",
ExpectError: true,
},
{
InputPath: ".",
ConfigurationPath: ".",
ExpectError: true,
},
{
InputPath: "",
ConfigurationPath: ".",
ExpectError: false,
},
{
InputPath: "api",
ConfigurationPath: ".",
ExpectError: false,
},
} {
orEmptyStr := func(s string) string {
if s == "" {
return "<empty>"
}
return s
}
t.Run(
fmt.Sprintf("with inputPath=%s, configurationPath=%s, expectError=%v",
orEmptyStr(tc.InputPath), orEmptyStr(tc.ConfigurationPath), tc.ExpectError),
func(t *testing.T) {
ds := &datasources.DataSource{
UID: "dsUID",
JsonData: simplejson.New(),
}
routes := []*plugins.Route{
{
Path: tc.ConfigurationPath,
ReqRole: org.RoleAdmin,
Method: "GET",
},
{
Path: tc.ConfigurationPath,
ReqRole: org.RoleAdmin,
Method: "POST",
},
{
Path: tc.ConfigurationPath,
ReqRole: org.RoleAdmin,
Method: "PUT",
},
{
Path: tc.ConfigurationPath,
ReqRole: org.RoleAdmin,
Method: "DELETE",
},
}
req, err := http.NewRequestWithContext(t.Context(), "GET", "http://localhost/"+tc.InputPath, nil)
require.NoError(t, err, "failed to create HTTP request")
ctx := &contextmodel.ReqContext{
Context: &web.Context{Req: req},
SignedInUser: &user.SignedInUser{OrgRole: org.RoleViewer},
}
proxy, err := setupDSProxyTest(t, ctx, ds, routes, tc.InputPath)
require.NoError(t, err, "failed to setup proxy test")
err = proxy.validateRequest()
if tc.ExpectError {
require.ErrorIs(t, err, errPluginProxyRouteAccessDenied, "request was not denied due to access denied?")
} else {
require.NoError(t, err, "request was unexpectedly denied access")
}
},
)
}
})
}
// test DataSourceProxy request handling.
+29 -73
View File
@@ -36,9 +36,6 @@ var client = &http.Client{
Transport: &http.Transport{Proxy: http.ProxyFromEnvironment},
}
// CreateDashboardSnapshot creates a snapshot when running Grafana in regular mode.
// It validates the user and dashboard exist before creating the snapshot.
// This mode supports both local and external snapshots.
func CreateDashboardSnapshot(c *contextmodel.ReqContext, cfg snapshot.SnapshotSharingOptions, cmd CreateDashboardSnapshotCommand, svc Service) {
if !cfg.SnapshotsEnabled {
c.JsonApiErr(http.StatusForbidden, "Dashboard Snapshots are disabled", nil)
@@ -46,7 +43,6 @@ func CreateDashboardSnapshot(c *contextmodel.ReqContext, cfg snapshot.SnapshotSh
}
uid := cmd.Dashboard.GetNestedString("uid")
user, err := identity.GetRequester(c.Req.Context())
if err != nil {
c.JsonApiErr(http.StatusBadRequest, "missing user in context", nil)
@@ -63,18 +59,21 @@ func CreateDashboardSnapshot(c *contextmodel.ReqContext, cfg snapshot.SnapshotSh
return
}
cmd.ExternalURL = ""
cmd.OrgID = user.GetOrgID()
cmd.UserID, _ = identity.UserIdentifier(user.GetID())
if cmd.Name == "" {
cmd.Name = "Unnamed snapshot"
}
var snapshotURL string
var snapshotUrl string
cmd.ExternalURL = ""
cmd.OrgID = user.GetOrgID()
cmd.UserID, _ = identity.UserIdentifier(user.GetID())
originalDashboardURL, err := createOriginalDashboardURL(&cmd)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Invalid app URL", err)
return
}
if cmd.External {
// Handle external snapshot creation
if !cfg.ExternalEnabled {
c.JsonApiErr(http.StatusForbidden, "External dashboard creation is disabled", nil)
return
@@ -86,83 +85,40 @@ func CreateDashboardSnapshot(c *contextmodel.ReqContext, cfg snapshot.SnapshotSh
return
}
snapshotUrl = resp.Url
cmd.Key = resp.Key
cmd.DeleteKey = resp.DeleteKey
cmd.ExternalURL = resp.Url
cmd.ExternalDeleteURL = resp.DeleteUrl
cmd.Dashboard = &common.Unstructured{}
snapshotURL = resp.Url
metrics.MApiDashboardSnapshotExternal.Inc()
} else {
// Handle local snapshot creation
originalDashboardURL, err := createOriginalDashboardURL(&cmd)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Invalid app URL", err)
return
cmd.Dashboard.SetNestedField(originalDashboardURL, "snapshot", "originalUrl")
if cmd.Key == "" {
var err error
cmd.Key, err = util.GetRandomString(32)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Could not generate random string", err)
return
}
}
snapshotURL, err = prepareLocalSnapshot(&cmd, originalDashboardURL)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Could not generate random string", err)
return
if cmd.DeleteKey == "" {
var err error
cmd.DeleteKey, err = util.GetRandomString(32)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Could not generate random string", err)
return
}
}
snapshotUrl = setting.ToAbsUrl("dashboard/snapshot/" + cmd.Key)
metrics.MApiDashboardSnapshotCreate.Inc()
}
saveAndRespond(c, svc, cmd, snapshotURL)
}
// CreateDashboardSnapshotPublic creates a snapshot when running Grafana in public mode.
// In public mode, there is no user or dashboard information to validate.
// Only local snapshots are supported (external snapshots are not available).
func CreateDashboardSnapshotPublic(c *contextmodel.ReqContext, cfg snapshot.SnapshotSharingOptions, cmd CreateDashboardSnapshotCommand, svc Service) {
if !cfg.SnapshotsEnabled {
c.JsonApiErr(http.StatusForbidden, "Dashboard Snapshots are disabled", nil)
return
}
if cmd.Name == "" {
cmd.Name = "Unnamed snapshot"
}
snapshotURL, err := prepareLocalSnapshot(&cmd, "")
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Could not generate random string", err)
return
}
metrics.MApiDashboardSnapshotCreate.Inc()
saveAndRespond(c, svc, cmd, snapshotURL)
}
// prepareLocalSnapshot prepares the command for a local snapshot and returns the snapshot URL.
func prepareLocalSnapshot(cmd *CreateDashboardSnapshotCommand, originalDashboardURL string) (string, error) {
cmd.Dashboard.SetNestedField(originalDashboardURL, "snapshot", "originalUrl")
if cmd.Key == "" {
key, err := util.GetRandomString(32)
if err != nil {
return "", err
}
cmd.Key = key
}
if cmd.DeleteKey == "" {
deleteKey, err := util.GetRandomString(32)
if err != nil {
return "", err
}
cmd.DeleteKey = deleteKey
}
return setting.ToAbsUrl("dashboard/snapshot/" + cmd.Key), nil
}
// saveAndRespond saves the snapshot and sends the response.
func saveAndRespond(c *contextmodel.ReqContext, svc Service, cmd CreateDashboardSnapshotCommand, snapshotURL string) {
result, err := svc.CreateDashboardSnapshot(c.Req.Context(), &cmd)
if err != nil {
c.JsonApiErr(http.StatusInternalServerError, "Failed to create snapshot", err)
@@ -172,7 +128,7 @@ func saveAndRespond(c *contextmodel.ReqContext, svc Service, cmd CreateDashboard
c.JSON(http.StatusOK, snapshot.DashboardCreateResponse{
Key: result.Key,
DeleteKey: result.DeleteKey,
URL: snapshotURL,
URL: snapshotUrl,
DeleteURL: setting.ToAbsUrl("api/snapshots-delete/" + result.DeleteKey),
})
}
+35 -331
View File
@@ -20,30 +20,40 @@ import (
"github.com/grafana/grafana/pkg/web"
)
func createTestDashboard(t *testing.T) *common.Unstructured {
t.Helper()
dashboard := &common.Unstructured{}
dashboardData := map[string]any{
"uid": "test-dashboard-uid",
"id": 123,
func TestCreateDashboardSnapshot_DashboardNotFound(t *testing.T) {
mockService := &MockService{}
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
ExternalEnabled: false,
}
dashboardBytes, _ := json.Marshal(dashboardData)
_ = json.Unmarshal(dashboardBytes, dashboard)
return dashboard
}
func createTestUser() *user.SignedInUser {
return &user.SignedInUser{
testUser := &user.SignedInUser{
UserID: 1,
OrgID: 1,
Login: "testuser",
Name: "Test User",
Email: "test@example.com",
}
}
dashboard := &common.Unstructured{}
dashboardData := map[string]interface{}{
"uid": "test-dashboard-uid",
"id": 123,
}
dashboardBytes, _ := json.Marshal(dashboardData)
_ = json.Unmarshal(dashboardBytes, dashboard)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test Snapshot",
},
}
mockService.On("ValidateDashboardExists", mock.Anything, int64(1), "test-dashboard-uid").
Return(dashboards.ErrDashboardNotFound)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
req = req.WithContext(identity.WithRequester(req.Context(), testUser))
func createReqContext(t *testing.T, req *http.Request, testUser *user.SignedInUser) (*contextmodel.ReqContext, *httptest.ResponseRecorder) {
t.Helper()
recorder := httptest.NewRecorder()
ctx := &contextmodel.ReqContext{
Context: &web.Context{
@@ -53,319 +63,13 @@ func createReqContext(t *testing.T, req *http.Request, testUser *user.SignedInUs
SignedInUser: testUser,
Logger: log.NewNopLogger(),
}
return ctx, recorder
}
// TestCreateDashboardSnapshot tests snapshot creation in regular mode (non-public instance).
// These tests cover scenarios when Grafana is running as a regular server with user authentication.
func TestCreateDashboardSnapshot(t *testing.T) {
t.Run("should return error when dashboard not found", func(t *testing.T) {
mockService := &MockService{}
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
ExternalEnabled: false,
}
testUser := createTestUser()
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test Snapshot",
},
}
mockService.On("ValidateDashboardExists", mock.Anything, int64(1), "test-dashboard-uid").
Return(dashboards.ErrDashboardNotFound)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
req = req.WithContext(identity.WithRequester(req.Context(), testUser))
ctx, recorder := createReqContext(t, req, testUser)
CreateDashboardSnapshot(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusBadRequest, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "Dashboard not found", response["message"])
})
t.Run("should create external snapshot when external is enabled", func(t *testing.T) {
externalServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "/api/snapshots", r.URL.Path)
assert.Equal(t, "POST", r.Method)
response := map[string]any{
"key": "external-key",
"deleteKey": "external-delete-key",
"url": "https://external.example.com/dashboard/snapshot/external-key",
"deleteUrl": "https://external.example.com/api/snapshots-delete/external-delete-key",
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(response)
}))
defer externalServer.Close()
mockService := NewMockService(t)
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
ExternalEnabled: true,
ExternalSnapshotURL: externalServer.URL,
}
testUser := createTestUser()
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test External Snapshot",
External: true,
},
}
mockService.On("ValidateDashboardExists", mock.Anything, int64(1), "test-dashboard-uid").
Return(nil)
mockService.On("CreateDashboardSnapshot", mock.Anything, mock.Anything).
Return(&DashboardSnapshot{
Key: "external-key",
DeleteKey: "external-delete-key",
}, nil)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
req = req.WithContext(identity.WithRequester(req.Context(), testUser))
ctx, recorder := createReqContext(t, req, testUser)
CreateDashboardSnapshot(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusOK, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "external-key", response["key"])
assert.Equal(t, "external-delete-key", response["deleteKey"])
assert.Equal(t, "https://external.example.com/dashboard/snapshot/external-key", response["url"])
})
t.Run("should return forbidden when external is disabled", func(t *testing.T) {
mockService := NewMockService(t)
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
ExternalEnabled: false,
}
testUser := createTestUser()
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test External Snapshot",
External: true,
},
}
mockService.On("ValidateDashboardExists", mock.Anything, int64(1), "test-dashboard-uid").
Return(nil)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
req = req.WithContext(identity.WithRequester(req.Context(), testUser))
ctx, recorder := createReqContext(t, req, testUser)
CreateDashboardSnapshot(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusForbidden, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "External dashboard creation is disabled", response["message"])
})
t.Run("should create local snapshot", func(t *testing.T) {
mockService := NewMockService(t)
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
}
testUser := createTestUser()
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test Local Snapshot",
},
Key: "local-key",
DeleteKey: "local-delete-key",
}
mockService.On("ValidateDashboardExists", mock.Anything, int64(1), "test-dashboard-uid").
Return(nil)
mockService.On("CreateDashboardSnapshot", mock.Anything, mock.Anything).
Return(&DashboardSnapshot{
Key: "local-key",
DeleteKey: "local-delete-key",
}, nil)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
req = req.WithContext(identity.WithRequester(req.Context(), testUser))
ctx, recorder := createReqContext(t, req, testUser)
CreateDashboardSnapshot(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusOK, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "local-key", response["key"])
assert.Equal(t, "local-delete-key", response["deleteKey"])
assert.Contains(t, response["url"], "dashboard/snapshot/local-key")
assert.Contains(t, response["deleteUrl"], "api/snapshots-delete/local-delete-key")
})
}
// TestCreateDashboardSnapshotPublic tests snapshot creation in public mode.
// These tests cover scenarios when Grafana is running as a public snapshot server
// where no user authentication or dashboard validation is required.
func TestCreateDashboardSnapshotPublic(t *testing.T) {
t.Run("should create local snapshot without user context", func(t *testing.T) {
mockService := NewMockService(t)
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: true,
}
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test Snapshot",
},
Key: "test-key",
DeleteKey: "test-delete-key",
}
mockService.On("CreateDashboardSnapshot", mock.Anything, mock.Anything).
Return(&DashboardSnapshot{
Key: "test-key",
DeleteKey: "test-delete-key",
}, nil)
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
recorder := httptest.NewRecorder()
ctx := &contextmodel.ReqContext{
Context: &web.Context{
Req: req,
Resp: web.NewResponseWriter("POST", recorder),
},
Logger: log.NewNopLogger(),
}
CreateDashboardSnapshotPublic(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusOK, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "test-key", response["key"])
assert.Equal(t, "test-delete-key", response["deleteKey"])
assert.Contains(t, response["url"], "dashboard/snapshot/test-key")
assert.Contains(t, response["deleteUrl"], "api/snapshots-delete/test-delete-key")
})
t.Run("should return forbidden when snapshots are disabled", func(t *testing.T) {
mockService := NewMockService(t)
cfg := snapshot.SnapshotSharingOptions{
SnapshotsEnabled: false,
}
dashboard := createTestDashboard(t)
cmd := CreateDashboardSnapshotCommand{
DashboardCreateCommand: snapshot.DashboardCreateCommand{
Dashboard: dashboard,
Name: "Test Snapshot",
},
}
req, _ := http.NewRequest("POST", "/api/snapshots", nil)
recorder := httptest.NewRecorder()
ctx := &contextmodel.ReqContext{
Context: &web.Context{
Req: req,
Resp: web.NewResponseWriter("POST", recorder),
},
Logger: log.NewNopLogger(),
}
CreateDashboardSnapshotPublic(ctx, cfg, cmd, mockService)
assert.Equal(t, http.StatusForbidden, recorder.Code)
var response map[string]any
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "Dashboard Snapshots are disabled", response["message"])
})
}
// TestDeleteExternalDashboardSnapshot tests deletion of external snapshots.
// This function is called in public mode and doesn't require user context.
func TestDeleteExternalDashboardSnapshot(t *testing.T) {
t.Run("should return nil on successful deletion", func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
assert.Equal(t, "GET", r.Method)
w.WriteHeader(http.StatusOK)
}))
defer server.Close()
err := DeleteExternalDashboardSnapshot(server.URL)
assert.NoError(t, err)
})
t.Run("should gracefully handle already deleted snapshot", func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
response := map[string]any{
"message": "Failed to get dashboard snapshot",
}
_ = json.NewEncoder(w).Encode(response)
}))
defer server.Close()
err := DeleteExternalDashboardSnapshot(server.URL)
assert.NoError(t, err)
})
t.Run("should return error on unexpected status code", func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusNotFound)
}))
defer server.Close()
err := DeleteExternalDashboardSnapshot(server.URL)
assert.Error(t, err)
assert.Contains(t, err.Error(), "unexpected response when deleting external snapshot")
assert.Contains(t, err.Error(), "404")
})
t.Run("should return error on 500 with different message", func(t *testing.T) {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusInternalServerError)
response := map[string]any{
"message": "Some other error",
}
_ = json.NewEncoder(w).Encode(response)
}))
defer server.Close()
err := DeleteExternalDashboardSnapshot(server.URL)
assert.Error(t, err)
assert.Contains(t, err.Error(), "500")
})
CreateDashboardSnapshot(ctx, cfg, cmd, mockService)
mockService.AssertExpectations(t)
assert.Equal(t, http.StatusBadRequest, recorder.Code)
var response map[string]interface{}
err := json.Unmarshal(recorder.Body.Bytes(), &response)
require.NoError(t, err)
assert.Equal(t, "Dashboard not found", response["message"])
}
+3 -24
View File
@@ -14,7 +14,6 @@ import (
"github.com/grafana/grafana/pkg/apimachinery/validation"
"github.com/grafana/grafana/pkg/storage/unified/sql/db"
"github.com/grafana/grafana/pkg/storage/unified/sql/dbutil"
"github.com/grafana/grafana/pkg/storage/unified/sql/rvmanager"
"github.com/grafana/grafana/pkg/storage/unified/sql/sqltemplate"
gocache "github.com/patrickmn/go-cache"
)
@@ -869,18 +868,10 @@ func (d *dataStore) applyBackwardsCompatibleChanges(ctx context.Context, tx db.T
if key.Action == DataActionDeleted {
generation = 0
}
// In compatibility mode, the previous RV, when available, is saved as a microsecond
// timestamp, as is done in the SQL backend.
previousRV := event.PreviousRV
if event.PreviousRV > 0 && isSnowflake(event.PreviousRV) {
previousRV = rvmanager.RVFromSnowflake(event.PreviousRV)
}
_, err := dbutil.Exec(ctx, tx, sqlKVUpdateLegacyResourceHistory, sqlKVLegacyUpdateHistoryRequest{
SQLTemplate: sqltemplate.New(kv.dialect),
GUID: key.GUID,
PreviousRV: previousRV,
PreviousRV: event.PreviousRV,
Generation: generation,
})
@@ -909,7 +900,7 @@ func (d *dataStore) applyBackwardsCompatibleChanges(ctx context.Context, tx db.T
Name: key.Name,
Action: action,
Folder: key.Folder,
PreviousRV: previousRV,
PreviousRV: event.PreviousRV,
})
if err != nil {
@@ -925,7 +916,7 @@ func (d *dataStore) applyBackwardsCompatibleChanges(ctx context.Context, tx db.T
Name: key.Name,
Action: action,
Folder: key.Folder,
PreviousRV: previousRV,
PreviousRV: event.PreviousRV,
})
if err != nil {
@@ -947,15 +938,3 @@ func (d *dataStore) applyBackwardsCompatibleChanges(ctx context.Context, tx db.T
return nil
}
// isSnowflake returns whether the argument passed is a snowflake ID (new) or a microsecond timestamp (old).
// We try to interpret the number as a microsecond timestamp first. If it represents a time in the past,
// it is considered a microsecond timestamp. Snowflake IDs are much larger integers and would lead
// to dates in the future if interpreted as a microsecond timestamp.
func isSnowflake(rv int64) bool {
ts := time.UnixMicro(rv)
oneHourFromNow := time.Now().Add(time.Hour)
isMicroSecRV := ts.Before(oneHourFromNow)
return !isMicroSecRV
}
+4 -4
View File
@@ -184,9 +184,9 @@ func (n *eventStore) Get(ctx context.Context, key EventKey) (Event, error) {
}
// ListSince returns a sequence of events since the given resource version.
func (n *eventStore) ListKeysSince(ctx context.Context, sinceRV int64) iter.Seq2[string, error] {
func (n *eventStore) ListKeysSince(ctx context.Context, sinceRV int64, sortOrder SortOrder) iter.Seq2[string, error] {
opts := ListOptions{
Sort: SortOrderAsc,
Sort: sortOrder,
StartKey: fmt.Sprintf("%d", sinceRV),
}
return func(yield func(string, error) bool) {
@@ -202,9 +202,9 @@ func (n *eventStore) ListKeysSince(ctx context.Context, sinceRV int64) iter.Seq2
}
}
func (n *eventStore) ListSince(ctx context.Context, sinceRV int64) iter.Seq2[Event, error] {
func (n *eventStore) ListSince(ctx context.Context, sinceRV int64, sortOrder SortOrder) iter.Seq2[Event, error] {
return func(yield func(Event, error) bool) {
for evtKey, err := range n.ListKeysSince(ctx, sinceRV) {
for evtKey, err := range n.ListKeysSince(ctx, sinceRV, sortOrder) {
if err != nil {
yield(Event{}, err)
return
@@ -369,7 +369,7 @@ func testEventStoreListKeysSince(t *testing.T, ctx context.Context, store *event
// List events since RV 1500 (should get events with RV 2000 and 3000)
retrievedEvents := make([]string, 0, 2)
for eventKey, err := range store.ListKeysSince(ctx, 1500) {
for eventKey, err := range store.ListKeysSince(ctx, 1500, SortOrderAsc) {
require.NoError(t, err)
retrievedEvents = append(retrievedEvents, eventKey)
}
@@ -429,7 +429,7 @@ func testEventStoreListSince(t *testing.T, ctx context.Context, store *eventStor
// List events since RV 1500 (should get events with RV 2000 and 3000)
retrievedEvents := make([]Event, 0, 2)
for event, err := range store.ListSince(ctx, 1500) {
for event, err := range store.ListSince(ctx, 1500, SortOrderAsc) {
require.NoError(t, err)
retrievedEvents = append(retrievedEvents, event)
}
@@ -453,7 +453,7 @@ func TestEventStore_ListSince_Empty(t *testing.T) {
func testEventStoreListSinceEmpty(t *testing.T, ctx context.Context, store *eventStore) {
// List events when store is empty
retrievedEvents := make([]Event, 0)
for event, err := range store.ListSince(ctx, 0) {
for event, err := range store.ListSince(ctx, 0, SortOrderAsc) {
require.NoError(t, err)
retrievedEvents = append(retrievedEvents, event)
}
@@ -825,7 +825,7 @@ func testListKeysSinceWithSnowflakeTime(t *testing.T, ctx context.Context, store
// List events since 90 minutes ago using subtractDurationFromSnowflake
sinceRV := subtractDurationFromSnowflake(snowflakeFromTime(now), 90*time.Minute)
retrievedEvents := make([]string, 0)
for eventKey, err := range store.ListKeysSince(ctx, sinceRV) {
for eventKey, err := range store.ListKeysSince(ctx, sinceRV, SortOrderAsc) {
require.NoError(t, err)
retrievedEvents = append(retrievedEvents, eventKey)
}
@@ -842,7 +842,7 @@ func testListKeysSinceWithSnowflakeTime(t *testing.T, ctx context.Context, store
// List events since 30 minutes ago using subtractDurationFromSnowflake
sinceRV = subtractDurationFromSnowflake(snowflakeFromTime(now), 30*time.Minute)
retrievedEvents = make([]string, 0)
for eventKey, err := range store.ListKeysSince(ctx, sinceRV) {
for eventKey, err := range store.ListKeysSince(ctx, sinceRV, SortOrderAsc) {
require.NoError(t, err)
retrievedEvents = append(retrievedEvents, eventKey)
}
+1 -1
View File
@@ -119,7 +119,7 @@ func (n *pollingNotifier) Watch(ctx context.Context, opts watchOptions) <-chan E
return
case <-time.After(currentInterval):
foundEvents := false
for evt, err := range n.eventStore.ListSince(ctx, subtractDurationFromSnowflake(lastRV, opts.LookbackPeriod)) {
for evt, err := range n.eventStore.ListSince(ctx, subtractDurationFromSnowflake(lastRV, opts.LookbackPeriod), SortOrderAsc) {
if err != nil {
n.log.Error("Failed to list events since", "error", err)
continue
+15 -9
View File
@@ -456,27 +456,33 @@ func testNotifierWatchMultipleEvents(t *testing.T, ctx context.Context, notifier
},
}
errCh := make(chan error)
go func() {
for _, event := range testEvents {
errCh <- eventStore.Save(ctx, event)
err := eventStore.Save(ctx, event)
require.NoError(t, err)
}
}()
// Receive events
receivedEvents := make([]string, 0, len(testEvents))
for len(receivedEvents) != len(testEvents) {
receivedEvents := make([]Event, 0, len(testEvents))
for i := 0; i < len(testEvents); i++ {
select {
case event := <-events:
receivedEvents = append(receivedEvents, event.Name)
case err := <-errCh:
require.NoError(t, err)
receivedEvents = append(receivedEvents, event)
case <-time.After(1 * time.Second):
t.Fatalf("Timed out waiting for event %d", len(receivedEvents)+1)
t.Fatalf("Timed out waiting for event %d", i+1)
}
}
// Verify all events were received
assert.Len(t, receivedEvents, len(testEvents))
// Verify the events match and ordered by resource version
receivedNames := make([]string, len(receivedEvents))
for i, event := range receivedEvents {
receivedNames[i] = event.Name
}
expectedNames := []string{"test-resource-1", "test-resource-2", "test-resource-3"}
assert.ElementsMatch(t, expectedNames, receivedEvents)
assert.ElementsMatch(t, expectedNames, receivedNames)
}
+2
View File
@@ -473,6 +473,8 @@ func (k *sqlKV) Delete(ctx context.Context, section string, key string) error {
return ErrNotFound
}
// TODO reflect change to resource table
return nil
}
+22 -17
View File
@@ -347,7 +347,7 @@ func (k *kvStorageBackend) WriteEvent(ctx context.Context, event WriteEvent) (in
return 0, fmt.Errorf("failed to write data: %w", err)
}
rv = rvmanager.SnowflakeFromRV(rv)
rv = rvmanager.SnowflakeFromRv(rv)
dataKey.ResourceVersion = rv
} else {
err := k.dataStore.Save(ctx, dataKey, bytes.NewReader(event.Value))
@@ -801,8 +801,20 @@ func (k *kvStorageBackend) ListModifiedSince(ctx context.Context, key Namespaced
}
}
// Generate a new resource version for the list
listRV := k.snowflake.Generate().Int64()
latestEvent, err := k.eventStore.LastEventKey(ctx)
if err != nil {
if errors.Is(err, ErrNotFound) {
return sinceRv, func(yield func(*ModifiedResource, error) bool) { /* nothing to return */ }
}
return 0, func(yield func(*ModifiedResource, error) bool) {
yield(nil, fmt.Errorf("error trying to retrieve last event key: %s", err))
}
}
if latestEvent.ResourceVersion == sinceRv {
return sinceRv, func(yield func(*ModifiedResource, error) bool) { /* nothing to return */ }
}
// Check if sinceRv is older than 1 hour
sinceRvTimestamp := snowflake.ID(sinceRv).Time()
@@ -811,11 +823,11 @@ func (k *kvStorageBackend) ListModifiedSince(ctx context.Context, key Namespaced
if sinceRvAge > time.Hour {
k.log.Debug("ListModifiedSince using data store", "sinceRv", sinceRv, "sinceRvAge", sinceRvAge)
return listRV, k.listModifiedSinceDataStore(ctx, key, sinceRv)
return latestEvent.ResourceVersion, k.listModifiedSinceDataStore(ctx, key, sinceRv)
}
k.log.Debug("ListModifiedSince using event store", "sinceRv", sinceRv, "sinceRvAge", sinceRvAge)
return listRV, k.listModifiedSinceEventStore(ctx, key, sinceRv)
return latestEvent.ResourceVersion, k.listModifiedSinceEventStore(ctx, key, sinceRv)
}
func convertEventType(action DataAction) resourcepb.WatchEvent_Type {
@@ -916,9 +928,9 @@ func (k *kvStorageBackend) listModifiedSinceDataStore(ctx context.Context, key N
func (k *kvStorageBackend) listModifiedSinceEventStore(ctx context.Context, key NamespacedResource, sinceRv int64) iter.Seq2[*ModifiedResource, error] {
return func(yield func(*ModifiedResource, error) bool) {
// store all events ordered by RV for the given tenant here
eventKeys := make([]EventKey, 0)
for evtKeyStr, err := range k.eventStore.ListKeysSince(ctx, subtractDurationFromSnowflake(sinceRv, defaultLookbackPeriod)) {
// we only care about the latest revision of every resource in the list
seen := make(map[string]struct{})
for evtKeyStr, err := range k.eventStore.ListKeysSince(ctx, subtractDurationFromSnowflake(sinceRv, defaultLookbackPeriod), SortOrderDesc) {
if err != nil {
yield(&ModifiedResource{}, err)
return
@@ -938,18 +950,11 @@ func (k *kvStorageBackend) listModifiedSinceEventStore(ctx context.Context, key
continue
}
eventKeys = append(eventKeys, evtKey)
}
// we only care about the latest revision of every resource in the list
seen := make(map[string]struct{})
for i := len(eventKeys) - 1; i >= 0; i -= 1 {
evtKey := eventKeys[i]
if _, ok := seen[evtKey.Name]; ok {
continue
}
seen[evtKey.Name] = struct{}{}
seen[evtKey.Name] = struct{}{}
value, err := k.getValueFromDataStore(ctx, DataKey(evtKey))
if err != nil {
yield(&ModifiedResource{}, err)
@@ -1307,7 +1312,7 @@ func (b *kvStorageBackend) ProcessBulk(ctx context.Context, setting BulkSettings
if setting.RebuildCollection {
for _, key := range setting.Collection {
events := make([]string, 0)
for evtKeyStr, err := range b.eventStore.ListKeysSince(ctx, 1) {
for evtKeyStr, err := range b.eventStore.ListKeysSince(ctx, 1, SortOrderAsc) {
if err != nil {
b.log.Error("failed to list event: %s", err)
return rsp
@@ -307,7 +307,7 @@ func (m *ResourceVersionManager) execBatch(ctx context.Context, group, resource
// Allocate the RVs
for i, guid := range guids {
guidToRV[guid] = rv
guidToSnowflakeRV[guid] = SnowflakeFromRV(rv)
guidToSnowflakeRV[guid] = SnowflakeFromRv(rv)
rvs[i] = rv
rv++
}
@@ -364,20 +364,12 @@ func (m *ResourceVersionManager) execBatch(ctx context.Context, group, resource
}
}
// takes a unix microsecond RV and transforms into a snowflake format. The timestamp is converted from microsecond to
// takes a unix microsecond rv and transforms into a snowflake format. The timestamp is converted from microsecond to
// millisecond (the integer division) and the remainder is saved in the stepbits section. machine id is always 0
func SnowflakeFromRV(rv int64) int64 {
func SnowflakeFromRv(rv int64) int64 {
return (((rv / 1000) - snowflake.Epoch) << (snowflake.NodeBits + snowflake.StepBits)) + (rv % 1000)
}
// It is generally not possible to convert from a snowflakeID to a microsecond RV due to the loss in precision
// (snowflake ID stores timestamp in milliseconds). However, this implementation stores the microsecond fraction
// in the step bits (see SnowflakeFromRV), allowing us to compute the microsecond timestamp.
func RVFromSnowflake(snowflakeID int64) int64 {
microSecFraction := snowflakeID & ((1 << snowflake.StepBits) - 1)
return ((snowflakeID>>(snowflake.NodeBits+snowflake.StepBits))+snowflake.Epoch)*1000 + microSecFraction
}
// helper utility to compare two RVs. The first RV must be in snowflake format. Will convert rv2 to snowflake and retry
// if comparison fails
func IsRvEqual(rv1, rv2 int64) bool {
@@ -385,7 +377,7 @@ func IsRvEqual(rv1, rv2 int64) bool {
return true
}
return rv1 == SnowflakeFromRV(rv2)
return rv1 == SnowflakeFromRv(rv2)
}
// Lock locks the resource version for the given key
@@ -63,13 +63,3 @@ func TestResourceVersionManager(t *testing.T) {
require.Equal(t, rv, int64(200))
})
}
func TestSnowflakeFromRVRoundtrips(t *testing.T) {
// 2026-01-12 19:33:58.806211 +0000 UTC
offset := int64(1768246438806211) // in microseconds
for n := range int64(100) {
ts := offset + n
require.Equal(t, ts, RVFromSnowflake(SnowflakeFromRV(ts)))
}
}
+11 -6
View File
@@ -23,6 +23,7 @@ import (
"github.com/grafana/authlib/types"
"github.com/grafana/grafana/pkg/apimachinery/utils"
"github.com/grafana/grafana/pkg/infra/db"
"github.com/grafana/grafana/pkg/storage/unified/resource"
"github.com/grafana/grafana/pkg/storage/unified/resourcepb"
sqldb "github.com/grafana/grafana/pkg/storage/unified/sql/db"
@@ -99,6 +100,10 @@ func RunStorageBackendTest(t *testing.T, newBackend NewBackendFunc, opts *TestOp
}
t.Run(tc.name, func(t *testing.T) {
if db.IsTestDbSQLite() {
t.Skip("Skipping tests on sqlite until channel notifier is implemented")
}
tc.fn(t, newBackend(context.Background()), opts.NSPrefix)
})
}
@@ -550,7 +555,7 @@ func runTestIntegrationBackendListModifiedSince(t *testing.T, backend resource.S
Resource: "resource",
}
latestRv, seq := backend.ListModifiedSince(ctx, key, rvCreated)
require.Greater(t, latestRv, rvCreated)
require.Equal(t, latestRv, rvDeleted)
counter := 0
for res, err := range seq {
@@ -624,11 +629,11 @@ func runTestIntegrationBackendListModifiedSince(t *testing.T, backend resource.S
rvCreated3, _ := writeEvent(ctx, backend, "bItem", resourcepb.WatchEvent_ADDED, WithNamespace(ns))
latestRv, seq := backend.ListModifiedSince(ctx, key, rvCreated1-1)
require.Greater(t, latestRv, rvCreated3)
require.Equal(t, latestRv, rvCreated3)
counter := 0
names := []string{"aItem", "bItem", "cItem"}
rvs := []int64{rvCreated2, rvCreated3, rvCreated1}
names := []string{"bItem", "aItem", "cItem"}
rvs := []int64{rvCreated3, rvCreated2, rvCreated1}
for res, err := range seq {
require.NoError(t, err)
require.Equal(t, key.Namespace, res.Key.Namespace)
@@ -1166,7 +1171,7 @@ func runTestIntegrationBackendCreateNewResource(t *testing.T, backend resource.S
}))
server := newServer(t, backend)
ns := nsPrefix + "-create-resource"
ns := nsPrefix + "-create-rsrce" // create-resource
ctx = request.WithNamespace(ctx, ns)
request := &resourcepb.CreateRequest{
@@ -1607,7 +1612,7 @@ func (s *sliceBulkRequestIterator) RollbackRequested() bool {
func runTestIntegrationBackendOptimisticLocking(t *testing.T, backend resource.StorageBackend, nsPrefix string) {
ctx := testutil.NewTestContext(t, time.Now().Add(30*time.Second))
ns := nsPrefix + "-optimistic-locking"
ns := nsPrefix + "-optimis-lock" // optimistic-locking. need to cut down on characters to not exceed namespace character limit (40)
t.Run("concurrent updates with same RV - only one succeeds", func(t *testing.T) {
// Create initial resource with rv0 (no previous RV)
@@ -36,6 +36,10 @@ func NewTestSqlKvBackend(t *testing.T, ctx context.Context, withRvManager bool)
KvStore: kv,
}
if db.DriverName() == "sqlite3" {
kvOpts.UseChannelNotifier = true
}
if withRvManager {
dialect := sqltemplate.DialectForDriver(db.DriverName())
rvManager, err := rvmanager.NewResourceVersionManager(rvmanager.ResourceManagerOptions{
@@ -200,7 +204,7 @@ func verifyKeyPath(t *testing.T, db sqldb.DB, ctx context.Context, key *resource
var keyPathRV int64
if isSqlBackend {
// Convert microsecond RV to snowflake for key_path construction
keyPathRV = rvmanager.SnowflakeFromRV(resourceVersion)
keyPathRV = rvmanager.SnowflakeFromRv(resourceVersion)
} else {
// KV backend already provides snowflake RV
keyPathRV = resourceVersion
@@ -434,6 +438,9 @@ func verifyResourceHistoryTable(t *testing.T, db sqldb.DB, namespace string, res
rows, err := db.QueryContext(ctx, query, namespace)
require.NoError(t, err)
defer func() {
_ = rows.Close()
}()
var records []ResourceHistoryRecord
for rows.Next() {
@@ -457,34 +464,33 @@ func verifyResourceHistoryTable(t *testing.T, db sqldb.DB, namespace string, res
for resourceIdx, res := range resources {
// Check create record (action=1, generation=1)
createRecord := records[recordIndex]
verifyResourceHistoryRecord(t, createRecord, namespace, res, resourceIdx, 1, 0, 1, resourceVersions[resourceIdx][0])
verifyResourceHistoryRecord(t, createRecord, res, resourceIdx, 1, 0, 1, resourceVersions[resourceIdx][0])
recordIndex++
}
for resourceIdx, res := range resources {
// Check update record (action=2, generation=2)
updateRecord := records[recordIndex]
verifyResourceHistoryRecord(t, updateRecord, namespace, res, resourceIdx, 2, resourceVersions[resourceIdx][0], 2, resourceVersions[resourceIdx][1])
verifyResourceHistoryRecord(t, updateRecord, res, resourceIdx, 2, resourceVersions[resourceIdx][0], 2, resourceVersions[resourceIdx][1])
recordIndex++
}
for resourceIdx, res := range resources[:2] {
// Check delete record (action=3, generation=0) - only first 2 resources were deleted
deleteRecord := records[recordIndex]
verifyResourceHistoryRecord(t, deleteRecord, namespace, res, resourceIdx, 3, resourceVersions[resourceIdx][1], 0, resourceVersions[resourceIdx][2])
verifyResourceHistoryRecord(t, deleteRecord, res, resourceIdx, 3, resourceVersions[resourceIdx][1], 0, resourceVersions[resourceIdx][2])
recordIndex++
}
}
// verifyResourceHistoryRecord validates a single resource_history record
func verifyResourceHistoryRecord(t *testing.T, record ResourceHistoryRecord, namespace string, expectedRes struct{ name, folder string }, resourceIdx, expectedAction int, expectedPrevRV int64, expectedGeneration int, expectedRV int64) {
func verifyResourceHistoryRecord(t *testing.T, record ResourceHistoryRecord, expectedRes struct{ name, folder string }, resourceIdx, expectedAction int, expectedPrevRV int64, expectedGeneration int, expectedRV int64) {
// Validate GUID (should be non-empty)
require.NotEmpty(t, record.GUID, "GUID should not be empty")
// Validate group/resource/namespace/name
require.Equal(t, "playlist.grafana.app", record.Group)
require.Equal(t, "playlists", record.Resource)
require.Equal(t, namespace, record.Namespace)
require.Equal(t, expectedRes.name, record.Name)
// Validate value contains expected JSON - server modifies/formats the JSON differently for different operations
@@ -511,12 +517,8 @@ func verifyResourceHistoryRecord(t *testing.T, record ResourceHistoryRecord, nam
// For KV backend operations, expectedPrevRV is now in snowflake format (returned by KV backend)
// but resource_history table stores microsecond RV, so we need to use IsRvEqual for comparison
if strings.Contains(record.Namespace, "-kv") {
if expectedPrevRV == 0 {
require.Zero(t, record.PreviousResourceVersion)
} else {
require.Equal(t, expectedPrevRV, rvmanager.SnowflakeFromRV(record.PreviousResourceVersion),
"Previous resource version should match (KV backend snowflake format)")
}
require.True(t, rvmanager.IsRvEqual(expectedPrevRV, record.PreviousResourceVersion),
"Previous resource version should match (KV backend snowflake format)")
} else {
require.Equal(t, expectedPrevRV, record.PreviousResourceVersion)
}
@@ -548,6 +550,9 @@ func verifyResourceTable(t *testing.T, db sqldb.DB, namespace string, resources
rows, err := db.QueryContext(ctx, query, namespace)
require.NoError(t, err)
defer func() {
_ = rows.Close()
}()
var records []ResourceRecord
for rows.Next() {
@@ -611,6 +616,9 @@ func verifyResourceVersionTable(t *testing.T, db sqldb.DB, namespace string, res
// Check that we have exactly one entry for playlist.grafana.app/playlists
rows, err := db.QueryContext(ctx, query, "playlist.grafana.app", "playlists")
require.NoError(t, err)
defer func() {
_ = rows.Close()
}()
var records []ResourceVersionRecord
for rows.Next() {
@@ -645,7 +653,7 @@ func verifyResourceVersionTable(t *testing.T, db sqldb.DB, namespace string, res
isKvBackend := strings.Contains(namespace, "-kv")
recordResourceVersion := record.ResourceVersion
if isKvBackend {
recordResourceVersion = rvmanager.SnowflakeFromRV(record.ResourceVersion)
recordResourceVersion = rvmanager.SnowflakeFromRv(record.ResourceVersion)
}
require.Less(t, recordResourceVersion, int64(9223372036854775807), "resource_version should be reasonable")
@@ -837,20 +845,24 @@ func runMixedConcurrentOperations(t *testing.T, sqlServer, kvServer resource.Res
}
// SQL backend operations
wg.Go(func() {
wg.Add(1)
go func() {
defer wg.Done()
<-startBarrier // Wait for signal to start
if err := runBackendOperationsWithCounts(ctx, sqlServer, namespace+"-sql", "sql", opCounts); err != nil {
errors <- fmt.Errorf("SQL backend operations failed: %w", err)
}
})
}()
// KV backend operations
wg.Go(func() {
wg.Add(1)
go func() {
defer wg.Done()
<-startBarrier // Wait for signal to start
if err := runBackendOperationsWithCounts(ctx, kvServer, namespace+"-kv", "kv", opCounts); err != nil {
errors <- fmt.Errorf("KV backend operations failed: %w", err)
}
})
}()
// Start both goroutines simultaneously
close(startBarrier)
@@ -30,7 +30,6 @@ func TestBadgerKVStorageBackend(t *testing.T) {
SkipTests: map[string]bool{
// TODO: fix these tests and remove this skip
TestBlobSupport: true,
TestListModifiedSince: true,
// Badger does not support bulk import yet.
TestGetResourceLastImportTime: true,
},
@@ -41,17 +40,8 @@ func TestIntegrationSQLKVStorageBackend(t *testing.T) {
testutil.SkipIntegrationTestInShortMode(t)
skipTests := map[string]bool{
TestWatchWriteEvents: true,
TestList: true,
TestBlobSupport: true,
TestGetResourceStats: true,
TestListHistory: true,
TestListHistoryErrorReporting: true,
TestListModifiedSince: true,
TestListTrash: true,
TestCreateNewResource: true,
TestGetResourceLastImportTime: true,
TestOptimisticLocking: true,
}
t.Run("Without RvManager", func(t *testing.T) {
@@ -59,7 +49,7 @@ func TestIntegrationSQLKVStorageBackend(t *testing.T) {
backend, _ := NewTestSqlKvBackend(t, ctx, false)
return backend
}, &TestOptions{
NSPrefix: "sqlkvstorage-test",
NSPrefix: "sqlkvstoragetest",
SkipTests: skipTests,
})
})
@@ -69,7 +59,7 @@ func TestIntegrationSQLKVStorageBackend(t *testing.T) {
backend, _ := NewTestSqlKvBackend(t, ctx, true)
return backend
}, &TestOptions{
NSPrefix: "sqlkvstorage-withrvmanager-test",
NSPrefix: "sqlkvstoragetest-rvmanager",
SkipTests: skipTests,
})
})
+39 -62
View File
@@ -30,7 +30,6 @@ const (
defaultLogGroupLimit = int32(50)
logIdentifierInternal = "__log__grafana_internal__"
logStreamIdentifierInternal = "__logstream__grafana_internal__"
logGroupsMacro = "$__logGroups"
)
type AWSError struct {
@@ -190,47 +189,6 @@ func (ds *DataSource) executeStartQuery(ctx context.Context, logsClient models.C
logsQuery.QueryLanguage = &cwli
}
region := logsQuery.Region
if region == "" || region == defaultRegion {
region = ds.Settings.Region
}
useARN := false
if len(logsQuery.LogGroups) > 0 && features.IsEnabled(ctx, features.FlagCloudWatchCrossAccountQuerying) && region != "" {
isMonitoringAccount, err := ds.isMonitoringAccount(ctx, region)
if err != nil {
ds.logger.FromContext(ctx).Debug("failed to determine monitoring account status", "err", err)
} else {
useARN = isMonitoringAccount
}
}
var logGroupIdentifiers []string
if len(logsQuery.LogGroups) > 0 {
// Log queries should use ARNs when querying a monitoring account because log group names are not unique across accounts.
if useARN {
for _, lg := range logsQuery.LogGroups {
if lg.Arn != "" {
// The startQuery api does not support arns with a trailing * so we need to remove it
logGroupIdentifiers = append(logGroupIdentifiers, strings.TrimSuffix(lg.Arn, "*"))
}
}
} else {
// deduplicate log group names because we only deduplicate log groups by their ARNs instead of their names when the query is created
seen := make(map[string]struct{}, len(logsQuery.LogGroups))
for _, lg := range logsQuery.LogGroups {
if lg.Name == "" {
continue
}
if _, exists := seen[lg.Name]; exists {
continue
}
seen[lg.Name] = struct{}{}
logGroupIdentifiers = append(logGroupIdentifiers, lg.Name)
}
}
}
finalQueryString := logsQuery.QueryString
// Only for CWLI queries
// The fields @log and @logStream are always included in the results of a user's query
@@ -242,21 +200,6 @@ func (ds *DataSource) executeStartQuery(ctx context.Context, logsClient models.C
logStreamIdentifierInternal + "|" + logsQuery.QueryString
}
// Expand $__logGroups macro for SQL queries
if *logsQuery.QueryLanguage == dataquery.LogsQueryLanguageSQL {
if strings.Contains(finalQueryString, logGroupsMacro) {
if len(logGroupIdentifiers) == 0 {
return nil, backend.DownstreamError(fmt.Errorf("query contains %s but no log groups are selected", logGroupsMacro))
}
quoted := make([]string, len(logGroupIdentifiers))
for i, id := range logGroupIdentifiers {
quoted[i] = fmt.Sprintf("'%s'", id)
}
replacement := fmt.Sprintf("`logGroups(logGroupIdentifier: [%s])`", strings.Join(quoted, ", "))
finalQueryString = strings.Replace(finalQueryString, logGroupsMacro, replacement, 1)
}
}
startQueryInput := &cloudwatchlogs.StartQueryInput{
StartTime: aws.Int64(startTime.Unix()),
// Usually grafana time range allows only second precision, but you can create ranges with milliseconds
@@ -270,13 +213,47 @@ func (ds *DataSource) executeStartQuery(ctx context.Context, logsClient models.C
// log group identifiers can be left out if the query is an SQL query
if *logsQuery.QueryLanguage != dataquery.LogsQueryLanguageSQL {
if useARN {
startQueryInput.LogGroupIdentifiers = logGroupIdentifiers
} else {
useLogGroupIdentifiers := false
logGroupsFromQuery := len(logsQuery.LogGroups) > 0
if logGroupsFromQuery && features.IsEnabled(ctx, features.FlagCloudWatchCrossAccountQuerying) {
region := logsQuery.Region
if region == "" || region == defaultRegion {
region = ds.Settings.Region
}
if region != "" {
isMonitoringAccount, err := ds.isMonitoringAccount(ctx, region)
if err != nil {
ds.logger.FromContext(ctx).Debug("failed to determine monitoring account status", "err", err)
} else if isMonitoringAccount {
// monitoring accounts require querying by log group identifiers because log group names are not unique across accounts.
var logGroupIdentifiers []string
for _, lg := range logsQuery.LogGroups {
// due to a bug in the startQuery api, we remove * from the arn, otherwise it throws an error
arn := strings.TrimSuffix(lg.Arn, "*")
logGroupIdentifiers = append(logGroupIdentifiers, arn)
}
startQueryInput.LogGroupIdentifiers = logGroupIdentifiers
useLogGroupIdentifiers = true
}
}
}
if !useLogGroupIdentifiers {
// even though logsQuery.LogGroupNames is deprecated, we still need to support it for backwards compatibility and alert queries
startQueryInput.LogGroupNames = append([]string(nil), logsQuery.LogGroupNames...)
if len(startQueryInput.LogGroupNames) == 0 && len(logGroupIdentifiers) > 0 {
startQueryInput.LogGroupNames = logGroupIdentifiers
if len(startQueryInput.LogGroupNames) == 0 && logGroupsFromQuery {
// deduplicate log group names because we only deduplicate log groups by their ARNs instead of their names when the query is created
seenLogGroupNames := make(map[string]struct{}, len(logsQuery.LogGroups))
for _, lg := range logsQuery.LogGroups {
if lg.Name == "" {
continue
}
if _, exists := seenLogGroupNames[lg.Name]; exists {
continue
}
seenLogGroupNames[lg.Name] = struct{}{}
startQueryInput.LogGroupNames = append(startQueryInput.LogGroupNames, lg.Name)
}
}
}
}
-198
View File
@@ -873,204 +873,6 @@ func TestQuery_GetQueryResults(t *testing.T) {
}, resp)
}
func Test_expandLogGroupsMacro(t *testing.T) {
origNewCWLogsClient := NewCWLogsClient
t.Cleanup(func() {
NewCWLogsClient = origNewCWLogsClient
})
var cli fakeCWLogsClient
NewCWLogsClient = func(cfg aws.Config) models.CWLogsClient {
return &cli
}
t.Run("expands $__logGroups macro with log group names when not a monitoring account", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource()
_, err := ds.QueryData(context.Background(), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM $__logGroups",
"logGroups":[{"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group1", "name": "group1"}, {"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group2", "name": "group2"}]
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Equal(t, "SELECT * FROM `logGroups(logGroupIdentifier: ['group1', 'group2'])`", *cli.calls.startQuery[0].QueryString)
})
t.Run("expands $__logGroups macro with ARNs when monitoring account", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource(func(ds *DataSource) {
ds.monitoringAccountCache.Store("us-east-1", true)
})
_, err := ds.QueryData(contextWithFeaturesEnabled(features.FlagCloudWatchCrossAccountQuerying), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM $__logGroups",
"logGroups":[{"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group1", "name": "group1"}, {"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group2", "name": "group2"}],
"region": "us-east-1"
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Equal(t, "SELECT * FROM `logGroups(logGroupIdentifier: ['arn:aws:logs:us-east-1:123456789012:log-group:group1', 'arn:aws:logs:us-east-1:123456789012:log-group:group2'])`", *cli.calls.startQuery[0].QueryString)
})
t.Run("strips trailing * from ARNs when expanding macro", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource(func(ds *DataSource) {
ds.monitoringAccountCache.Store("us-east-1", true)
})
_, err := ds.QueryData(contextWithFeaturesEnabled(features.FlagCloudWatchCrossAccountQuerying), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM $__logGroups",
"logGroups":[{"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group1*", "name": "group1"}],
"region": "us-east-1"
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Equal(t, "SELECT * FROM `logGroups(logGroupIdentifier: ['arn:aws:logs:us-east-1:123456789012:log-group:group1'])`", *cli.calls.startQuery[0].QueryString)
})
t.Run("returns error when $__logGroups macro is used but no log groups are selected", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource()
resp, err := ds.QueryData(context.Background(), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM $__logGroups"
}`),
},
},
})
assert.NoError(t, err)
assert.Contains(t, resp.Responses["A"].Error.Error(), "query contains $__logGroups but no log groups are selected")
})
t.Run("does not expand macro when query does not contain $__logGroups", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource()
_, err := ds.QueryData(context.Background(), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM ` + "`logGroups(logGroupIdentifier: ['my-log-group'])`" + `"
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Equal(t, "SELECT * FROM `logGroups(logGroupIdentifier: ['my-log-group'])`", *cli.calls.startQuery[0].QueryString)
})
t.Run("does not expand macro for non-SQL query languages", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource()
_, err := ds.QueryData(context.Background(), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "CWLI",
"queryString":"fields @message | $__logGroups",
"logGroups":[{"arn": "arn:aws:logs:us-east-1:123456789012:log-group:group1", "name": "group1"}]
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Contains(t, *cli.calls.startQuery[0].QueryString, "$__logGroups")
})
t.Run("expands macro with single log group", func(t *testing.T) {
cli = fakeCWLogsClient{}
ds := newTestDatasource()
_, err := ds.QueryData(context.Background(), &backend.QueryDataRequest{
PluginContext: backend.PluginContext{DataSourceInstanceSettings: &backend.DataSourceInstanceSettings{}},
Queries: []backend.DataQuery{
{
RefID: "A",
TimeRange: backend.TimeRange{From: time.Unix(0, 0), To: time.Unix(1, 0)},
JSON: json.RawMessage(`{
"type": "logAction",
"subtype": "StartQuery",
"queryLanguage": "SQL",
"queryString":"SELECT * FROM $__logGroups",
"logGroups":[{"arn": "arn:aws:logs:us-east-1:123456789012:log-group:single-group", "name": "single-group"}]
}`),
},
},
})
assert.NoError(t, err)
require.Len(t, cli.calls.startQuery, 1)
assert.Equal(t, "SELECT * FROM `logGroups(logGroupIdentifier: ['single-group'])`", *cli.calls.startQuery[0].QueryString)
})
}
func TestGroupResponseFrame(t *testing.T) {
t.Run("Doesn't group results without time field", func(t *testing.T) {
frame := data.NewFrameOfFieldTypes("test", 0, data.FieldTypeString, data.FieldTypeInt32)
@@ -36,7 +36,7 @@ export const DEFAULT_ANNOTATIONS_QUERY: Omit<CloudWatchAnnotationQuery, 'refId'>
export const DEFAULT_CWLI_QUERY_STRING = 'fields @timestamp, @message |\nsort @timestamp desc |\nlimit 20';
export const DEFAULT_PPL_QUERY_STRING = 'fields `@timestamp`, `@message`\n| sort - `@timestamp`\n| head 25s';
export const DEFAULT_SQL_QUERY_STRING =
'SELECT `@timestamp`, `@message`\nFROM $__logGroups\nORDER BY `@timestamp` DESC\nLIMIT 25;';
'SELECT `@timestamp`, `@message`\nFROM `log_group`\nORDER BY `@timestamp` DESC\nLIMIT 25;';
export const getDefaultLogsQuery = (
defaultLogGroups?: LogGroup[],
@@ -97,22 +97,14 @@ describe('LogsSQLCompletionItemProvider', () => {
const suggestions = await getSuggestions(singleLineFullQuery.query, { lineNumber: 1, column: 103 });
const suggestionLabels = suggestions.map((s) => s.label);
expect(suggestionLabels).toEqual(
expect.arrayContaining([
FROM,
`${FROM} $__logGroups`,
`${FROM} \`logGroups(logGroupIdentifier: [...])\``,
CASE,
...ALL_FUNCTIONS,
])
expect.arrayContaining([FROM, `${FROM} \`logGroups(logGroupIdentifier: [...])\``, CASE, ...ALL_FUNCTIONS])
);
});
it('returns logGroups and $__logGroups suggestion after from keyword', async () => {
it('returns logGroups suggestion after from keyword', async () => {
const suggestions = await getSuggestions(singleLineFullQuery.query, { lineNumber: 1, column: 108 });
const suggestionLabels = suggestions.map((s) => s.label);
expect(suggestionLabels).toEqual(
expect.arrayContaining(['$__logGroups', '`logGroups(logGroupIdentifier: [...])`'])
);
expect(suggestionLabels).toEqual(expect.arrayContaining(['`logGroups(logGroupIdentifier: [...])`']));
});
it('returns where, having, limit, group by, order by, and join suggestions after from arguments', async () => {
@@ -142,12 +142,6 @@ export class LogsSQLCompletionItemProvider extends CompletionItemProvider {
command: TRIGGER_SUGGEST,
sortText: CompletionItemPriority.MediumHigh,
});
addSuggestion(`${FROM} $__logGroups`, {
insertText: `${FROM} $__logGroups`,
kind: monaco.languages.CompletionItemKind.Snippet,
sortText: CompletionItemPriority.High,
detail: 'Use selected log groups from the selector',
});
addSuggestion(`${FROM} \`logGroups(logGroupIdentifier: [...])\``, {
insertText: `${FROM} \`logGroups(logGroupIdentifier: [$0])\``,
insertTextRules: monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet,
@@ -158,12 +152,6 @@ export class LogsSQLCompletionItemProvider extends CompletionItemProvider {
break;
case SuggestionKind.AfterFromKeyword:
addSuggestion('$__logGroups', {
insertText: '$__logGroups',
kind: monaco.languages.CompletionItemKind.Variable,
sortText: CompletionItemPriority.High,
detail: 'Expands to selected log groups',
});
addSuggestion('`logGroups(logGroupIdentifier: [...])`', {
insertText: '`logGroups(logGroupIdentifier: [$0])`',
insertTextRules: monaco.languages.CompletionItemInsertTextRule.InsertAsSnippet,
@@ -488,7 +488,6 @@ export const language: CloudWatchLanguage = {
root: [
{ include: '@comments' },
{ include: '@whitespace' },
{ include: '@macros' },
{ include: '@customParams' },
{ include: '@numbers' },
{ include: '@binaries' },
@@ -520,7 +519,6 @@ export const language: CloudWatchLanguage = {
[/\*\//, { token: 'comment.quote', next: '@pop' }],
[/./, 'comment'],
],
macros: [[/\$__[a-zA-Z0-9_]+/, 'type']],
customParams: [
[/\${[A-Za-z0-9._-]*}/, 'variable'],
[/\@\@{[A-Za-z0-9._-]*}/, 'variable'],