Compare commits

...
Author SHA1 Message Date
Sven Grossmann f377d8b992 Merge branch 'main' into svennergr/drawer-sidebar-aware 2025-11-24 10:53:05 +01:00
Sven Grossmann a89078e8ee revert comments 2025-11-24 10:49:35 +01:00
Sven Grossmann aac0b71a22 Drawer: Cleanup 2025-11-24 10:45:37 +01:00
Sven Grossmann 28c83d5f47 AppChrome: Make page container relative 2025-11-24 10:44:11 +01:00
Zoltán Bedi 8d75d79313 PostgreSQL: Remove feature toggle postgresDSUsePGX (#113675)
* PostgreSQL: Remove feature toggle `postgresDSUsePGX`

* Fix tests and linting

* Address review comments
2025-11-24 10:26:41 +01:00
Hugo Häggmark 39dc659ad8 chore: reduce Azure barrel files (#114297) 2025-11-24 06:03:45 +01:00
grafana-pr-automation[bot]andgithub-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> 4b0d6a2e1a I18n: Download translations from Crowdin (#114332)
New Crowdin translations by GitHub Action

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-24 00:32:34 +00:00
Adela Almasan 2ee9561920 Chore: Fix panels smoke test (#113187)
* update test

* update
2025-11-21 20:35:25 +00:00
Larissa Wandzura 01eee72c80 DOCS: Fixed PDC link for CloudWatch data source (#114200)
fixed broken link (refURI) for PDC
2025-11-21 14:22:38 -06:00
Moustafa Baiou 8108d3c795 Alerting: Add tracing to prometheus rules api (#114282)
To better observe and identify performance bottlenecks in the prometheus rules API.

The following spans were added:
- `api.prometheus.RouteGetRuleStatuses`
- `api.prometheus.PrepareRuleGroupStatusesV2`

The `api.prometheus.PrepareRuleGroupStatusesV2` includes attributes to capture the parameters used in the request to help with debugging and performance analysis.
2025-11-21 15:00:35 -05:00
Adela Almasan 8cf777b5e8 VizSuggestions: Empty state (#114049)
* newVizSuggestions feature toggle

* panel empty state

* add data check

* select first suggestion

* update text

* PanelEmptyState component

* add test

* ?

* remove fake translation

* updates

* move empty state to UnconfiguredPanel
2025-11-21 13:42:41 -05:00
Denis Vodopianov 0e460a267e chore : Deprecating FeatureToggles.IsEnabled (#113062)
* Deprecating features.IsEnabled

* add one more nolint

* add one more nolint

* Give better hints to devs in the deprecation message of IsEnabledGlobally

* adding more doc strings

* fix linter after rebase

* Extend deprecation message
2025-11-21 18:43:42 +01:00
Matias Chomicki 43217025cf processing: set log clone as not collapsed (#114313) 2025-11-21 17:01:04 +00:00
Daniele Stefano Ferru 7299651a32 Hotfix: Granting None role Viewer access for a fixed API group list (#114310) 2025-11-21 17:50:25 +01:00
Beverly Buchanan 1b2fe6d6a8 RBAC plugin doc: add k8s rbac to list (#114265) 2025-11-21 17:44:00 +01:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Sven Grossmann 006af44285 Drawer: Make Drawer open to the left of ExtensionSidebar 2025-11-21 14:27:42 +01:00
161 changed files with 2575 additions and 5541 deletions
@@ -29,7 +29,7 @@ refs:
- pattern: /docs/grafana/
destination: docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/#cloud-access-policies-action-definitions
- pattern: /docs/grafana-cloud/
destination: /docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/#cloud-access-policies-action-definitions
destination: /docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/custom-role-actions-scopes/#create-access-policies
rbac-role-definitions:
- pattern: /docs/grafana/
destination: /docs/grafana/<GRAFANA_VERSION>/administration/roles-and-permissions/access-control/rbac-fixed-basic-role-definitions/
@@ -66,16 +66,17 @@ Please refer to plugin documentation to see what RBAC permissions the plugin has
The following list contains app plugins that have fine-grained RBAC support.
| App plugin | App plugin ID | App plugin permission documentation |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Access policies](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) | `grafana-auth-app` | [RBAC actions for Access Policies](ref:cloud-access-policies-action-definitions) |
| [Adaptive metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/adaptive-metrics-plugin/) | `grafana-adaptive-metrics-app` | [RBAC actions for Adaptive Metrics](ref:adaptive-metrics-permissions) |
| [Incident](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/incident/) | `grafana-incident-app` | n/a |
| [OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/) | `grafana-oncall-app` | [Configure RBAC for OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/manage/user-and-team-management/#manage-users-and-teams-for-grafana-oncall) |
| [Performance Testing (K6)](https://grafana.com/docs/grafana-cloud/testing/k6/) | `k6-app` | [Configure RBAC for K6](https://grafana.com/docs/grafana-cloud/testing/k6/projects-and-users/configure-rbac/) |
| [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) | `grafana-pdc-app` | n/a |
| [Service Level Objective (SLO)](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/) | `grafana-slo-app` | [Configure RBAC for SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/rbac/) |
| [Cloud Provider](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/) | `grafana-csp-app` | [Cloud Provider Observability role-based access control](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/rbac/) |
| App plugin | App plugin ID | App plugin permission documentation |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| [Access policies](https://grafana.com/docs/grafana-cloud/account-management/authentication-and-permissions/access-policies/) | `grafana-auth-app` | [RBAC actions for Access Policies](ref:cloud-access-policies-action-definitions) |
| [Adaptive Metrics](https://grafana.com/docs/grafana-cloud/cost-management-and-billing/reduce-costs/metrics-costs/control-metrics-usage-via-adaptive-metrics/adaptive-metrics-plugin/) | `grafana-adaptive-metrics-app` | [RBAC actions for Adaptive Metrics](ref:adaptive-metrics-permissions) |
| [Cloud Provider](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/) | `grafana-csp-app` | [Cloud Provider Observability role-based access control](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/monitor-cloud-provider/rbac/) |
| [Incident](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/incident/) | `grafana-incident-app` | n/a |
| [Kubernetes Monitoring](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/) | `grafana-k8s-app` | [Kubernetes Monitoring role-based access control](/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/control-access/#precision-access-with-rbac-custom-plugin-roles) |
| [OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/) | `grafana-oncall-app` | [Configure RBAC for OnCall](https://grafana.com/docs/grafana-cloud/alerting-and-irm/irm/oncall/manage/user-and-team-management/#manage-users-and-teams-for-grafana-oncall) |
| [Performance Testing (K6)](https://grafana.com/docs/grafana-cloud/testing/k6/) | `k6-app` | [Configure RBAC for K6](https://grafana.com/docs/grafana-cloud/testing/k6/projects-and-users/configure-rbac/) |
| [Private data source connect (PDC)](https://grafana.com/docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/) | `grafana-pdc-app` | n/a |
| [Service Level Objective (SLO)](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/) | `grafana-slo-app` | [Configure RBAC for SLO](https://grafana.com/docs/grafana-cloud/alerting-and-irm/slo/set-up/rbac/) |
### Revoke fine-grained access from app plugins
@@ -62,9 +62,9 @@ refs:
destination: /docs/grafana/<GRAFANA_VERSION>/datasources/aws-CloudWatch/aws-authentication/
private-data-source-connect:
- pattern: /docs/grafana/
destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
- pattern: /docs/grafana-cloud/
destination: docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/
configure-pdc:
- pattern: /docs/grafana/
destination: /docs/grafana-cloud/connect-externally-hosted/private-data-source-connect/configure-pdc/#configure-grafana-private-data-source-connect-pdc
+20 -10
View File
@@ -7,13 +7,13 @@ test.describe(
tag: ['@acceptance'],
},
() => {
test.skip('Tests each panel type in the panel edit view to ensure no crash', async ({
test('Tests each panel type in the panel edit view to ensure no crash', async ({
gotoDashboardPage,
selectors,
page,
}) => {
// this test can absolutely take longer than the default 30s timeout
test.setTimeout(60000);
test.setTimeout(120000);
// Create new dashboard
const dashboardPage = await gotoDashboardPage({});
@@ -29,22 +29,32 @@ test.describe(
return win.grafanaBootData?.settings?.panels ?? {};
});
const vizPicker = dashboardPage.getByGrafanaSelector(selectors.components.PanelEditor.toggleVizPicker);
// Loop through every panel type and ensure no crash
for (const [_, panel] of Object.entries(panelTypes)) {
if (panel.hideFromList || panel.state === 'deprecated') {
continue; // Skip hidden and deprecated panels
}
// Select the panel type in the viz picker
const vizPicker = dashboardPage.getByGrafanaSelector(selectors.components.PanelEditor.toggleVizPicker);
await vizPicker.click();
await dashboardPage.getByGrafanaSelector(selectors.components.PluginVisualization.item(panel.name)).click();
try {
// Select the panel type in the viz picker
await expect(vizPicker).toBeVisible();
await vizPicker.click({ force: true });
// Verify panel type is selected
await expect(vizPicker).toHaveText(panel.name);
await dashboardPage.getByGrafanaSelector(selectors.components.PluginVisualization.item(panel.name)).click();
// Ensure no unexpected error occurred
await expect(page.getByText('An unexpected error happened')).toBeHidden();
// Verify panel type is selected
await expect(vizPicker).toHaveText(panel.name, { timeout: 10000 });
// Wait for panel to finish rendering
await expect(page.getByLabel('Panel loading bar')).toHaveCount(0, { timeout: 10000 });
// Ensure no unexpected error occurred
await expect(page.getByText('An unexpected error happened')).toBeHidden();
} catch (error) {
throw new Error(`Panel '${panel.name}' failed: ${error}`);
}
}
});
}
-10
View File
@@ -3706,16 +3706,6 @@
"count": 7
}
},
"public/app/plugins/datasource/azuremonitor/types/query.ts": {
"no-barrel-files/no-barrel-files": {
"count": 3
}
},
"public/app/plugins/datasource/azuremonitor/types/templateVariables.ts": {
"no-barrel-files/no-barrel-files": {
"count": 1
}
},
"public/app/plugins/datasource/azuremonitor/utils/common.ts": {
"@typescript-eslint/consistent-type-assertions": {
"count": 1
-4
View File
@@ -922,10 +922,6 @@ export interface FeatureToggles {
*/
grafanaAssistantInProfilesDrilldown?: boolean;
/**
* Enables using PGX instead of libpq for PostgreSQL datasource
*/
postgresDSUsePGX?: boolean;
/**
* Enables creating alerts from Tempo data source
*/
tempoAlerting?: boolean;
@@ -107,7 +107,7 @@ export function Drawer({
open={true}
onClose={onClose}
placement="right"
getContainer={'.main-view'}
getContainer={'main'}
className={styles.drawerContent}
rootClassName={styles.drawer}
classNames={{
@@ -264,7 +264,10 @@ const getStyles = (theme: GrafanaTheme2) => {
position: 'relative',
}),
drawer: css({
top: 0,
// Override rc-drawer's default position: fixed to position: absolute
// so it positions relative to its container (.page-panes) instead of viewport
// eslint-disable-next-line @typescript-eslint/consistent-type-assertions
position: 'absolute !important' as 'absolute',
'.rc-drawer-content-wrapper': {
boxShadow: theme.shadows.z3,
@@ -306,6 +309,7 @@ const getStyles = (theme: GrafanaTheme2) => {
backgroundColor: 'transparent !important',
// eslint-disable-next-line @typescript-eslint/consistent-type-assertions
position: 'fixed !important' as 'fixed',
zIndex: theme.zIndex.modal - 1, // Below drawer (modal = 1060) and sidebar (modal = 1060)
'&:before': {
backgroundColor: `${theme.components.overlay.background} !important`,
@@ -313,8 +317,9 @@ const getStyles = (theme: GrafanaTheme2) => {
content: '""',
left: 0,
position: 'fixed',
// Start below the top nav bar
top: '40px',
right: 0,
top: 0,
},
}),
maskMotion: css({
@@ -372,6 +377,7 @@ function getWrapperStyles(theme: GrafanaTheme2, size: 'sm' | 'md' | 'lg') {
return css({
label: `drawer-content-wrapper-${size}`,
overflow: 'unset !important',
zIndex: theme.zIndex.modal, // Ensure drawer wrapper is above mask
[theme.breakpoints.down('md')]: {
width: `calc(100% - ${theme.spacing(2)}) !important`,
+2
View File
@@ -313,6 +313,7 @@ func (hs *HTTPServer) declareFixedRoles() error {
Grants: []string{string(org.RoleEditor)},
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(context.Background(), featuremgmt.FlagAnnotationPermissionUpdate) {
// Keeping the name to avoid breaking changes (for users who have assigned this role to grant permissions on organization annotations)
annotationsReaderRole = ac.RoleRegistration{
@@ -619,6 +620,7 @@ func (hs *HTTPServer) declareFixedRoles() error {
libraryPanelsReaderRole, libraryPanelsWriterRole, libraryPanelsGeneralReaderRole, libraryPanelsGeneralWriterRole,
snapshotsCreatorRole, snapshotsDeleterRole, snapshotsReaderRole}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(context.Background(), featuremgmt.FlagAnnotationPermissionUpdate) {
allAnnotationsReaderRole := ac.RoleRegistration{
Role: ac.RoleDTO{
+9
View File
@@ -126,6 +126,7 @@ func (hs *HTTPServer) PostAnnotation(c *contextmodel.ReqContext) response.Respon
}
if canSave, err := hs.canCreateAnnotation(c, cmd.DashboardUID); err != nil || !canSave {
//nolint:staticcheck // not yet migrated to OpenFeature
if !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
return dashboardGuardianResponse(err)
} else if err != nil {
@@ -271,6 +272,7 @@ func (hs *HTTPServer) UpdateAnnotation(c *contextmodel.ReqContext) response.Resp
return resp
}
//nolint:staticcheck // not yet migrated to OpenFeature
if !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
if canSave, err := hs.canSaveAnnotation(c, hs.AccessControl, annotation); err != nil || !canSave {
return dashboardGuardianResponse(err)
@@ -329,6 +331,7 @@ func (hs *HTTPServer) PatchAnnotation(c *contextmodel.ReqContext) response.Respo
return resp
}
//nolint:staticcheck // not yet migrated to OpenFeature
if !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
if canSave, err := hs.canSaveAnnotation(c, hs.AccessControl, annotation); err != nil || !canSave {
return dashboardGuardianResponse(err)
@@ -439,6 +442,7 @@ func (hs *HTTPServer) MassDeleteAnnotations(c *contextmodel.ReqContext) response
canSave, err := hs.canMassDeleteAnnotations(c, dashboardUID)
if err != nil || !canSave {
//nolint:staticcheck // not yet migrated to OpenFeature
if !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
return dashboardGuardianResponse(err)
} else if err != nil {
@@ -500,6 +504,7 @@ func (hs *HTTPServer) DeleteAnnotationByID(c *contextmodel.ReqContext) response.
return response.Error(http.StatusBadRequest, "annotationId is invalid", err)
}
//nolint:staticcheck // not yet migrated to OpenFeature
if !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
annotation, resp := findAnnotationByID(c.Req.Context(), hs.annotationsRepo, annotationID, c.SignedInUser)
if resp != nil {
@@ -610,6 +615,7 @@ func AnnotationTypeScopeResolver(annotationsRepo annotations.Repository, feature
},
}
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(ctx, featuremgmt.FlagAnnotationPermissionUpdate) {
tempUser = &user.SignedInUser{
OrgID: orgID,
@@ -626,6 +632,7 @@ func AnnotationTypeScopeResolver(annotationsRepo annotations.Repository, feature
return nil, errors.New("could not resolve annotation type")
}
//nolint:staticcheck // not yet migrated to OpenFeature
if !features.IsEnabled(ctx, featuremgmt.FlagAnnotationPermissionUpdate) {
switch annotation.GetType() {
case annotations.Organization:
@@ -662,6 +669,7 @@ func AnnotationTypeScopeResolver(annotationsRepo annotations.Repository, feature
}
func (hs *HTTPServer) canCreateAnnotation(c *contextmodel.ReqContext, dashboardUID string) (bool, error) {
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
if dashboardUID != "" {
evaluator := accesscontrol.EvalPermission(accesscontrol.ActionAnnotationsCreate, dashboards.ScopeDashboardsProvider.GetResourceScopeUID(dashboardUID))
@@ -686,6 +694,7 @@ func (hs *HTTPServer) canCreateAnnotation(c *contextmodel.ReqContext, dashboardU
}
func (hs *HTTPServer) canMassDeleteAnnotations(c *contextmodel.ReqContext, dashboardUID string) (bool, error) {
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagAnnotationPermissionUpdate) {
if dashboardUID == "" {
evaluator := accesscontrol.EvalPermission(accesscontrol.ActionAnnotationsDelete, accesscontrol.ScopeAnnotationsTypeOrganization)
+1
View File
@@ -166,6 +166,7 @@ func (hs *HTTPServer) GetDashboard(c *contextmodel.ReqContext) response.Response
}
annotationPermissions := &dashboardsV1.AnnotationPermission{}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(ctx, featuremgmt.FlagAnnotationPermissionUpdate) {
hs.getAnnotationPermissionsByScope(c, &annotationPermissions.Dashboard, dashboards.ScopeDashboardsProvider.GetResourceScopeUID(dash.UID))
} else {
+4
View File
@@ -150,6 +150,7 @@ func (hs *HTTPServer) getFrontendSettings(c *contextmodel.ReqContext) (*dtos.Fro
continue
}
//nolint:staticcheck // not yet migrated to OpenFeature
if panel.ID == "datagrid" && !hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagEnableDatagridEditing) {
continue
}
@@ -190,6 +191,7 @@ func (hs *HTTPServer) getFrontendSettings(c *contextmodel.ReqContext) (*dtos.Fro
hasAccess := accesscontrol.HasAccess(hs.AccessControl, c)
trustedTypesDefaultPolicyEnabled := (hs.Cfg.CSPEnabled && strings.Contains(hs.Cfg.CSPTemplate, "require-trusted-types-for")) || (hs.Cfg.CSPReportOnlyEnabled && strings.Contains(hs.Cfg.CSPReportOnlyTemplate, "require-trusted-types-for"))
//nolint:staticcheck // not yet migrated to OpenFeature
isCloudMigrationTarget := hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagOnPremToCloudMigrations) && hs.Cfg.CloudMigration.IsTarget
featureToggles := hs.Features.GetEnabled(c.Req.Context())
// this is needed for backwards compatibility with external plugins
@@ -406,6 +408,7 @@ func (hs *HTTPServer) getFrontendSettings(c *contextmodel.ReqContext) (*dtos.Fro
DisableSignoutMenu: hs.Cfg.DisableSignoutMenu,
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Cfg.PasswordlessMagicLinkAuth.Enabled && hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagPasswordlessMagicLinkAuthentication) {
hasEnabledProviders := hs.samlEnabled() || hs.authnService.IsClientEnabled(authn.ClientLDAP)
@@ -444,6 +447,7 @@ func (hs *HTTPServer) getFrontendSettings(c *contextmodel.ReqContext) (*dtos.Fro
frontendSettings.Namespace = hs.namespacer(c.OrgID)
// experimental scope features
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagScopeFilters) {
frontendSettings.ListScopesEndpoint = hs.Cfg.ScopesListScopesURL
frontendSettings.ListDashboardScopesEndpoint = hs.Cfg.ScopesListDashboardsURL
+2
View File
@@ -62,6 +62,7 @@ func (hs *HTTPServer) setIndexViewData(c *contextmodel.ReqContext) (*dtos.IndexV
return nil, err
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagIndividualCookiePreferences) {
if !prefs.Cookies("analytics") {
settings.GoogleAnalytics4Id = ""
@@ -94,6 +95,7 @@ func (hs *HTTPServer) setIndexViewData(c *contextmodel.ReqContext) (*dtos.IndexV
}
var regionalFormat string
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagLocaleFormatPreference) {
regionalFormat = "en"
+1
View File
@@ -361,6 +361,7 @@ func (hs *HTTPServer) RedirectResponseWithError(c *contextmodel.ReqContext, err
func (hs *HTTPServer) redirectURLWithErrorCookie(c *contextmodel.ReqContext, err error) string {
setCookie := true
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagIndividualCookiePreferences) {
var userID int64
if c.SignedInUser != nil && !c.IsNil() {
+1
View File
@@ -81,6 +81,7 @@ func (proxy *PluginProxy) HandleRequest() {
hasSlash := strings.HasSuffix(proxy.proxyPath, "/")
proxy.proxyPath = path
//nolint:staticcheck // not yet migrated to OpenFeature
if hasSlash && !strings.HasSuffix(path, "/") && proxy.features.IsEnabled(proxy.ctx.Req.Context(), featuremgmt.FlagPluginProxyPreserveTrailingSlash) {
proxy.proxyPath += "/"
}
+2
View File
@@ -144,6 +144,7 @@ func (hs *HTTPServer) GetPluginList(c *contextmodel.ReqContext) response.Respons
AngularDetected: pluginDef.Angular.Detected,
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Cfg.ManagedServiceAccountsEnabled && hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagExternalServiceAccounts) {
listItem.IAM = pluginDef.IAM
}
@@ -490,6 +491,7 @@ func (hs *HTTPServer) InstallPlugin(c *contextmodel.ReqContext) response.Respons
return response.ErrOrFallback(http.StatusInternalServerError, "Failed to install plugin", err)
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hs.Cfg.ManagedServiceAccountsEnabled && hs.Features.IsEnabled(c.Req.Context(), featuremgmt.FlagExternalServiceAccounts) {
// This is a non-blocking function that verifies that the installer has
// the permissions that the plugin requests to have on Grafana.
+1
View File
@@ -29,6 +29,7 @@ func (c *ResultConverter) Convert(ctx context.Context,
}
var dt data.FrameType
//nolint:staticcheck // not yet migrated to OpenFeature
dt, useDataplane, _ := shouldUseDataplane(frames, logger, c.Features.IsEnabled(ctx, featuremgmt.FlagDisableSSEDataplane))
if useDataplane {
logger.Debug("Handling SSE data source query through dataplane", "datatype", dt)
+1
View File
@@ -70,6 +70,7 @@ func handleDataplaneFrames(ctx context.Context, tracer tracing.Tracer, features
case data.KindTimeSeries:
return handleDataplaneTimeseries(frames)
case data.KindNumeric:
//nolint:staticcheck // not yet migrated to OpenFeature
sortMetrics := !features.IsEnabled(ctx, featuremgmt.FlagDisableNumericMetricsSortingInExpressions)
return handleDataplaneNumeric(frames, sortMetrics)
default:
+1 -1
View File
@@ -68,7 +68,7 @@ type DataPipeline []Node
// map of the refId of the of each command
func (dp *DataPipeline) execute(c context.Context, now time.Time, s *Service) (mathexp.Vars, error) {
vars := make(mathexp.Vars)
//nolint:staticcheck // not yet migrated to OpenFeature
groupByDSFlag := s.features.IsEnabled(c, featuremgmt.FlagSseGroupByDatasource)
// Execute datasource nodes first, and grouped by datasource.
if groupByDSFlag {
+1
View File
@@ -64,6 +64,7 @@ func (l *loggerImpl) Middleware() web.Middleware {
// put the start time on context so we can measure it later.
r = r.WithContext(log.InitstartTime(r.Context(), time.Now()))
//nolint:staticcheck // not yet migrated to OpenFeature
if l.flags.IsEnabled(r.Context(), featuremgmt.FlagUnifiedRequestLog) {
r = r.WithContext(errutil.SetUnifiedLogging(r.Context()))
}
+1
View File
@@ -114,6 +114,7 @@ func RequestMetrics(features featuremgmt.FeatureToggles, cfg *setting.Cfg, promR
handler = "notfound"
} else {
// log requests where we could not identify handler so we can register them.
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(r.Context(), featuremgmt.FlagLogRequestsInstrumentedAsUnknown) {
log.Warn("request instrumented as unknown", "path", r.URL.Path, "status_code", status)
}
@@ -237,7 +237,7 @@ func NewPlugin(pluginID string, cfg *setting.Cfg, httpClientProvider *httpclient
case Tempo:
svc = tempo.ProvideService(httpClientProvider, tracer)
case PostgreSQL:
svc = postgres.ProvideService(cfg)
svc = postgres.ProvideService()
case MySQL:
svc = mysql.ProvideService()
case MSSQL:
@@ -90,3 +90,8 @@ func (d *directResourceClient) Watch(ctx context.Context, in *resourcepb.WatchRe
func (d *directResourceClient) BulkProcess(ctx context.Context, opts ...grpc.CallOption) (resourcepb.BulkStore_BulkProcessClient, error) {
return nil, fmt.Errorf("BulkProcess not supported with direct resource client")
}
// RebuildIndexes implements resource.ResourceClient.
func (b *directResourceClient) RebuildIndexes(ctx context.Context, req *resourcepb.RebuildIndexesRequest, opts ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error) {
return nil, fmt.Errorf("not implemented")
}
@@ -109,13 +109,14 @@ func ProvideMigratorDashboardAccessor(
features featuremgmt.FeatureToggles,
) MigrationDashboardAccessor {
return &dashboardSqlAccess{
sql: sql,
namespacer: claims.OrgNamespaceFormatter,
dashStore: nil, // not needed for migration
provisioning: provisioning,
dashboardPermissionSvc: nil, // not needed for migration
libraryPanelSvc: nil, // not needed for migration
accessControl: accessControl,
sql: sql,
namespacer: claims.OrgNamespaceFormatter,
dashStore: nil, // not needed for migration
provisioning: provisioning,
dashboardPermissionSvc: nil, // not needed for migration
libraryPanelSvc: nil, // not needed for migration
accessControl: accessControl,
//nolint:staticcheck // not yet migrated to OpenFeature
invalidDashboardParseFallbackEnabled: features.IsEnabled(context.Background(), featuremgmt.FlagScanRowInvalidDashboardParseFallbackEnabled),
}
}
@@ -132,14 +133,15 @@ func NewDashboardSQLAccess(sql legacysql.LegacyDatabaseProvider,
) *dashboardSqlAccess {
dashboardSearchClient := legacysearcher.NewDashboardSearchClient(dashStore, sorter)
return &dashboardSqlAccess{
sql: sql,
namespacer: namespacer,
dashStore: dashStore,
provisioning: provisioning,
dashboardSearchClient: *dashboardSearchClient,
dashboardPermissionSvc: dashboardPermissionSvc,
libraryPanelSvc: libraryPanelSvc,
accessControl: accessControl,
sql: sql,
namespacer: namespacer,
dashStore: dashStore,
provisioning: provisioning,
dashboardSearchClient: *dashboardSearchClient,
dashboardPermissionSvc: dashboardPermissionSvc,
libraryPanelSvc: libraryPanelSvc,
accessControl: accessControl,
//nolint:staticcheck // not yet migrated to OpenFeature
invalidDashboardParseFallbackEnabled: features.IsEnabled(context.Background(), featuremgmt.FlagScanRowInvalidDashboardParseFallbackEnabled),
}
}
@@ -1046,3 +1048,7 @@ func parseLibraryPanelRow(p panel) (dashboardV0.LibraryPanel, error) {
return item, nil
}
func (b *dashboardSqlAccess) RebuildIndexes(ctx context.Context, req *resourcepb.RebuildIndexesRequest) (*resourcepb.RebuildIndexesResponse, error) {
return nil, fmt.Errorf("not implemented")
}
@@ -33,9 +33,11 @@ func (b *DashboardsAPIBuilder) ValidateDashboardSpec(ctx context.Context, obj ru
case *v0.Dashboard:
errorOnSchemaMismatches = false // Never error for v0
case *v1.Dashboard:
//nolint:staticcheck // not yet migrated to OpenFeature
errorOnSchemaMismatches = !b.features.IsEnabled(ctx, featuremgmt.FlagDashboardDisableSchemaValidationV1)
case *v2alpha1.Dashboard:
case *v2beta1.Dashboard:
//nolint:staticcheck // not yet migrated to OpenFeature
errorOnSchemaMismatches = !b.features.IsEnabled(ctx, featuremgmt.FlagDashboardDisableSchemaValidationV2)
default:
return nil, fmt.Errorf("invalid dashboard type: %T", obj)
@@ -45,6 +47,7 @@ func (b *DashboardsAPIBuilder) ValidateDashboardSpec(ctx context.Context, obj ru
return nil, apierrors.NewBadRequest("Not supported: FieldValidationMode: Warn")
}
//nolint:staticcheck // not yet migrated to OpenFeature
alwaysLogSchemaValidationErrors := b.features.IsEnabled(ctx, featuremgmt.FlagDashboardSchemaValidationLogging)
var errors field.ErrorList
@@ -581,3 +581,8 @@ func (m *mockSearchClient) GetStats(ctx context.Context, in *resourcepb.Resource
func (m *mockSearchClient) Search(ctx context.Context, in *resourcepb.ResourceSearchRequest, opts ...grpc.CallOption) (*resourcepb.ResourceSearchResponse, error) {
return m.search, m.searchErr
}
// RebuildIndexes implements resourcepb.ResourceIndexClient.
func (m *mockSearchClient) RebuildIndexes(ctx context.Context, in *resourcepb.RebuildIndexesRequest, opts ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error) {
return nil, fmt.Errorf("not implemented")
}
@@ -8,6 +8,7 @@ import (
"net"
"net/http"
"strconv"
"strings"
"sync"
"testing"
"time"
@@ -175,6 +176,28 @@ func TestIntegrationDistributor(t *testing.T) {
}
})
t.Run("RebuildIndexes", func(t *testing.T) {
instanceResponseCount := make(map[string]int)
// simulate RebuildIndexes for a single namespace
testNamespace := testNamespaces[0]
req := &resourcepb.RebuildIndexesRequest{
Namespace: testNamespace,
Keys: []*resourcepb.ResourceKey{{
Namespace: testNamespace,
Group: "folder.grafana.app",
Resource: "folders",
}},
}
distributorRes := getDistributorResponse(t, req, distributorServer.resourceClient.RebuildIndexes, instanceResponseCount)
require.Nil(t, distributorRes.Error)
// assert all instances got the response by looking at the merged details
count := strings.Count(distributorRes.Details, "{instance:")
require.Equal(t, len(testServers), count)
})
var wg sync.WaitGroup
for _, testServer := range testServers {
wg.Add(1)
+2 -2
View File
@@ -400,7 +400,7 @@ func Initialize(ctx context.Context, cfg *setting.Cfg, opts Options, apiOpts api
prometheusService := prometheus.ProvideService(httpclientProvider)
tempoService := tempo.ProvideService(httpclientProvider, tracer)
testdatasourceService := testdatasource.ProvideService()
postgresService := postgres.ProvideService(cfg)
postgresService := postgres.ProvideService()
mysqlService := mysql.ProvideService()
mssqlService := mssql.ProvideService(cfg)
entityEventsService := store.ProvideEntityEventsService(cfg, sqlStore, featureToggles)
@@ -1047,7 +1047,7 @@ func InitializeForTest(ctx context.Context, t sqlutil.ITestDB, testingT interfac
prometheusService := prometheus.ProvideService(httpclientProvider)
tempoService := tempo.ProvideService(httpclientProvider, tracer)
testdatasourceService := testdatasource.ProvideService()
postgresService := postgres.ProvideService(cfg)
postgresService := postgres.ProvideService()
mysqlService := mysql.ProvideService()
mssqlService := mssql.ProvideService(cfg)
entityEventsService := store.ProvideEntityEventsService(cfg, sqlStore, featureToggles)
@@ -741,6 +741,7 @@ func (s *Service) SaveExternalServiceRole(ctx context.Context, cmd accesscontrol
ctx, span := tracer.Start(ctx, "accesscontrol.acimpl.SaveExternalServiceRole")
defer span.End()
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.cfg.ManagedServiceAccountsEnabled || !s.features.IsEnabled(ctx, featuremgmt.FlagExternalServiceAccounts) {
s.log.Debug("Registering an external service role is behind a feature flag, enable it to use this feature.")
return nil
@@ -757,6 +758,7 @@ func (s *Service) DeleteExternalServiceRole(ctx context.Context, externalService
ctx, span := tracer.Start(ctx, "accesscontrol.acimpl.DeleteExternalServiceRole")
defer span.End()
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.cfg.ManagedServiceAccountsEnabled || !s.features.IsEnabled(ctx, featuremgmt.FlagExternalServiceAccounts) {
s.log.Debug("Deleting an external service role is behind a feature flag, enable it to use this feature.")
return nil
@@ -28,6 +28,7 @@ var DashboardEditActions = append(DashboardViewActions, []string{dashboards.Acti
var DashboardAdminActions = append(DashboardEditActions, []string{dashboards.ActionDashboardsPermissionsRead, dashboards.ActionDashboardsPermissionsWrite}...)
func getDashboardViewActions(features featuremgmt.FeatureToggles) []string {
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(context.Background(), featuremgmt.FlagAnnotationPermissionUpdate) {
return append(DashboardViewActions, accesscontrol.ActionAnnotationsRead)
}
@@ -35,6 +36,7 @@ func getDashboardViewActions(features featuremgmt.FeatureToggles) []string {
}
func getDashboardEditActions(features featuremgmt.FeatureToggles) []string {
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(context.Background(), featuremgmt.FlagAnnotationPermissionUpdate) {
return append(DashboardEditActions, []string{accesscontrol.ActionAnnotationsRead, accesscontrol.ActionAnnotationsWrite, accesscontrol.ActionAnnotationsDelete, accesscontrol.ActionAnnotationsCreate}...)
}
@@ -42,6 +44,7 @@ func getDashboardEditActions(features featuremgmt.FeatureToggles) []string {
}
func getDashboardAdminActions(features featuremgmt.FeatureToggles) []string {
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(context.Background(), featuremgmt.FlagAnnotationPermissionUpdate) {
return append(DashboardAdminActions, []string{accesscontrol.ActionAnnotationsRead, accesscontrol.ActionAnnotationsWrite, accesscontrol.ActionAnnotationsDelete, accesscontrol.ActionAnnotationsCreate}...)
}
@@ -61,6 +61,7 @@ func (authz *AuthService) Authorize(ctx context.Context, query annotations.ItemQ
scopeTypes := annotationScopeTypes(scopes)
_, canAccessOrgAnnotations := scopeTypes[annotations.Organization.String()]
_, canAccessDashAnnotations := scopeTypes[annotations.Dashboard.String()]
//nolint:staticcheck // not yet migrated to OpenFeature
if authz.features.IsEnabled(ctx, featuremgmt.FlagAnnotationPermissionUpdate) {
canAccessDashAnnotations = true
}
@@ -122,6 +123,7 @@ func (authz *AuthService) dashboardsWithVisibleAnnotations(ctx context.Context,
}
filterType := searchstore.TypeDashboard
//nolint:staticcheck // not yet migrated to OpenFeature
if authz.features.IsEnabled(ctx, featuremgmt.FlagAnnotationPermissionUpdate) {
filterType = searchstore.TypeAnnotation
}
@@ -3,6 +3,7 @@ package authorizer
import (
"context"
"fmt"
"slices"
"k8s.io/apiserver/pkg/authorization/authorizer"
@@ -12,6 +13,10 @@ import (
var _ authorizer.Authorizer = &roleAuthorizer{}
var orgRoleNoneAsViewerAPIGroups = []string{
"productactivation.ext.grafana.com",
}
type roleAuthorizer struct{}
func newRoleAuthorizer() *roleAuthorizer {
@@ -43,6 +48,16 @@ func (auth roleAuthorizer) Authorize(ctx context.Context, a authorizer.Attribute
return authorizer.DecisionDeny, errorMessageForGrafanaOrgRole(orgRole, a), nil
}
case org.RoleNone:
// HOTFIX: granting Viewer actions to None roles to a fixed group of APIs,
// while we work on a proper fix.
if slices.Contains(orgRoleNoneAsViewerAPIGroups, a.GetAPIGroup()) {
switch a.GetVerb() {
case "get", "list", "watch":
return authorizer.DecisionAllow, "", nil
default:
return authorizer.DecisionDeny, errorMessageForGrafanaOrgRole(orgRole, a), nil
}
}
return authorizer.DecisionDeny, errorMessageForGrafanaOrgRole(orgRole, a), nil
}
return authorizer.DecisionDeny, "", nil
@@ -83,6 +83,7 @@ func ProvideRegistration(
}
}
//nolint:staticcheck // not yet migrated to OpenFeature
if cfg.PasswordlessMagicLinkAuth.Enabled && features.IsEnabled(context.Background(), featuremgmt.FlagPasswordlessMagicLinkAuthentication) {
hasEnabledProviders := authnSvc.IsClientEnabled(authn.ClientSAML) || authnSvc.IsClientEnabled(authn.ClientLDAP)
if !hasEnabledProviders {
+1
View File
@@ -210,6 +210,7 @@ func (c *CachingServiceClient) WithQueryDataCaching(ctx context.Context, req *ba
// Update the query cache with the result for this metrics request
if err == nil && cr.UpdateCacheFn != nil {
// If AWS async caching is not enabled, use the old code path
//nolint:staticcheck // not yet migrated to OpenFeature
if c.features == nil || !c.features.IsEnabled(ctx, featuremgmt.FlagAwsAsyncQueryCaching) {
cr.UpdateCacheFn(ctx, resp)
} else if reqCtx != nil {
+7 -1
View File
@@ -10,6 +10,10 @@ import (
type FeatureToggles interface {
// IsEnabled checks if a feature is enabled for a given context.
// The settings may be per user, tenant, or globally set in the cloud
//
// Deprecated: FeatureToggles.IsEnabled is deprecated and will be removed in a future release.
// Evaluate with OpenFeature instead (see [github.com/open-feature/go-sdk/openfeature.Client]), for example:
// openfeature.NewDefaultClient().Boolean(ctx, "your-flag", false, openfeature.TransactionContext(ctx))
IsEnabled(ctx context.Context, flag string) bool
// IsEnabledGlobally checks if a flag is configured globally. For now, this is the same
@@ -19,7 +23,9 @@ type FeatureToggles interface {
// a full server restart for a change to take place.
//
// Deprecated: FeatureToggles.IsEnabledGlobally is deprecated and will be removed in a future release.
// Evaluate with OpenFeature instead (see [github.com/open-feature/go-sdk/openfeature.Client])
// Toggles that must be reliably evaluated at the service startup should be
// changed to settings (see setting.StartupSettings), and/or removed entirely.
// For app registration please use `grafana-apiserver.runtime_config` in settings.ini
IsEnabledGlobally(flag string) bool
// Get the enabled flags -- this *may* also include disabled flags (with value false)
-6
View File
@@ -1518,12 +1518,6 @@ var (
FrontendOnly: true,
Expression: "true",
},
{
Name: "postgresDSUsePGX",
Description: "Enables using PGX instead of libpq for PostgreSQL datasource",
Stage: FeatureStageExperimental,
Owner: grafanaOSSBigTent,
},
{
Name: "tempoAlerting",
Description: "Enables creating alerts from Tempo data source",
-1
View File
@@ -208,7 +208,6 @@ unifiedNavbars,GA,@grafana/plugins-platform-backend,false,false,true
logsPanelControls,preview,@grafana/observability-logs,false,false,true
metricsFromProfiles,experimental,@grafana/observability-traces-and-profiling,false,false,true
grafanaAssistantInProfilesDrilldown,GA,@grafana/observability-traces-and-profiling,false,false,true
postgresDSUsePGX,experimental,@grafana/oss-big-tent,false,false,false
tempoAlerting,experimental,@grafana/observability-traces-and-profiling,false,false,false
pluginsAutoUpdate,experimental,@grafana/plugins-platform-backend,false,false,false
alertingListViewV2PreviewToggle,privatePreview,@grafana/alerting-squad,false,false,true
1 Name Stage Owner requiresDevMode RequiresRestart FrontendOnly
208 logsPanelControls preview @grafana/observability-logs false false true
209 metricsFromProfiles experimental @grafana/observability-traces-and-profiling false false true
210 grafanaAssistantInProfilesDrilldown GA @grafana/observability-traces-and-profiling false false true
postgresDSUsePGX experimental @grafana/oss-big-tent false false false
211 tempoAlerting experimental @grafana/observability-traces-and-profiling false false false
212 pluginsAutoUpdate experimental @grafana/plugins-platform-backend false false false
213 alertingListViewV2PreviewToggle privatePreview @grafana/alerting-squad false false true
-4
View File
@@ -606,10 +606,6 @@ const (
// use multi-tenant path for awsTempCredentials
FlagMultiTenantTempCredentials = "multiTenantTempCredentials"
// FlagPostgresDSUsePGX
// Enables using PGX instead of libpq for PostgreSQL datasource
FlagPostgresDSUsePGX = "postgresDSUsePGX"
// FlagTempoAlerting
// Enables creating alerts from Tempo data source
FlagTempoAlerting = "tempoAlerting"
+1
View File
@@ -148,6 +148,7 @@ func (l *LibraryElementService) deleteHandler(c *contextmodel.ReqContext) respon
// 404: notFoundError
// 500: internalServerError
func (l *LibraryElementService) getHandler(c *contextmodel.ReqContext) response.Response {
//nolint:staticcheck // not yet migrated to OpenFeature
if l.features.IsEnabled(c.Req.Context(), featuremgmt.FlagKubernetesLibraryPanels) {
l.k8sHandler.getK8sLibraryElement(c)
return nil // already handled in the k8s handler
@@ -44,6 +44,7 @@ func (s *ServiceImpl) getAdminNode(c *contextmodel.ReqContext) (*navtree.NavLink
Text: "Organizations", SubTitle: "Isolated instances of Grafana running on the same server", Id: "global-orgs", Url: s.cfg.AppSubURL + "/admin/orgs", Icon: "building",
})
}
//nolint:staticcheck // not yet migrated to OpenFeature
if hasAccess(cloudmigration.MigrationAssistantAccess) && s.features.IsEnabled(ctx, featuremgmt.FlagOnPremToCloudMigrations) {
generalNodeLinks = append(generalNodeLinks, &navtree.NavLink{
Text: "Migrate to Grafana Cloud",
@@ -99,6 +100,7 @@ func (s *ServiceImpl) getAdminNode(c *contextmodel.ReqContext) (*navtree.NavLink
})
}
//nolint:staticcheck // not yet migrated to OpenFeature
if (s.cfg.Env == setting.Dev) || s.features.IsEnabled(ctx, featuremgmt.FlagEnableExtensionsAdminPage) && hasAccess(pluginaccesscontrol.AdminAccessEvaluator) {
pluginsNodeLinks = append(pluginsNodeLinks, &navtree.NavLink{
Text: "Extensions",
@@ -147,6 +149,7 @@ func (s *ServiceImpl) getAdminNode(c *contextmodel.ReqContext) (*navtree.NavLink
})
}
//nolint:staticcheck // not yet migrated to OpenFeature
if s.license.FeatureEnabled("groupsync") &&
s.features.IsEnabled(ctx, featuremgmt.FlagGroupAttributeSync) &&
hasAccess(ac.EvalAny(
@@ -407,6 +407,7 @@ func (s *ServiceImpl) buildDashboardNavLinks(c *contextmodel.ReqContext) []*navt
})
}
//nolint:staticcheck // not yet migrated to OpenFeature
if s.features.IsEnabled(c.Req.Context(), featuremgmt.FlagRestoreDashboards) && (c.GetOrgRole() == org.RoleAdmin || c.IsGrafanaAdmin) {
dashboardChildNavs = append(dashboardChildNavs, &navtree.NavLink{
Text: "Recently deleted",
@@ -435,6 +436,7 @@ func (s *ServiceImpl) buildAlertNavLinks(c *contextmodel.ReqContext) *navtree.Na
hasAccess := ac.HasAccess(s.accessControl, c)
var alertChildNavs []*navtree.NavLink
//nolint:staticcheck // not yet migrated to OpenFeature
if s.features.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingTriage) {
if hasAccess(ac.EvalAny(ac.EvalPermission(ac.ActionAlertingRuleRead), ac.EvalPermission(ac.ActionAlertingRuleExternalRead))) {
alertChildNavs = append(alertChildNavs, &navtree.NavLink{
@@ -492,6 +494,7 @@ func (s *ServiceImpl) buildAlertNavLinks(c *contextmodel.ReqContext) *navtree.Na
alertChildNavs = append(alertChildNavs, &navtree.NavLink{Text: "Alert groups", SubTitle: "See grouped alerts with active notifications", Id: "groups", Url: s.cfg.AppSubURL + "/alerting/groups", Icon: "layer-group"})
}
//nolint:staticcheck // not yet migrated to OpenFeature
if s.features.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingCentralAlertHistory) {
if hasAccess(ac.EvalAny(ac.EvalPermission(ac.ActionAlertingRuleRead))) {
alertChildNavs = append(alertChildNavs, &navtree.NavLink{
@@ -503,6 +506,7 @@ func (s *ServiceImpl) buildAlertNavLinks(c *contextmodel.ReqContext) *navtree.Na
})
}
}
//nolint:staticcheck // not yet migrated to OpenFeature
if c.GetOrgRole() == org.RoleAdmin && s.features.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertRuleRestore) && s.features.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingRuleRecoverDeleted) {
alertChildNavs = append(alertChildNavs, &navtree.NavLink{
Text: "Recently deleted",
@@ -77,6 +77,7 @@ func (srv ConfigSrv) RoutePostNGalertConfig(c *contextmodel.ReqContext, body api
return response.Error(http.StatusBadRequest, "Invalid alertmanager choice specified", err)
}
//nolint:staticcheck // not yet migrated to OpenFeature
disableExternal := srv.featureManager.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingDisableSendAlertsExternal)
if disableExternal && sendAlertsTo != ngmodels.InternalAlertmanager {
return response.Error(http.StatusBadRequest, "Sending alerts to external alertmanagers is disallowed on this instance", err)
+3
View File
@@ -79,6 +79,7 @@ func (srv TestingApiSrv) RouteTestGrafanaRuleConfig(c *contextmodel.ReqContext,
return response.ErrOrFallback(http.StatusInternalServerError, "failed to authorize access to rule group", err)
}
//nolint:staticcheck // not yet migrated to OpenFeature
if srv.featureManager.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingQueryOptimization) {
if _, err := store.OptimizeAlertQueries(rule.Data); err != nil {
return ErrResp(http.StatusInternalServerError, err, "Failed to optimize query")
@@ -178,6 +179,7 @@ func (srv TestingApiSrv) RouteEvalQueries(c *contextmodel.ReqContext, cmd apimod
}
var optimizations []store.Optimization
//nolint:staticcheck // not yet migrated to OpenFeature
if srv.featureManager.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingQueryOptimization) {
var err error
optimizations, err = store.OptimizeAlertQueries(cond.Data)
@@ -223,6 +225,7 @@ func addOptimizedQueryWarnings(evalResults *backend.QueryDataResponse, optimizat
}
func (srv TestingApiSrv) BacktestAlertRule(c *contextmodel.ReqContext, cmd apimodels.BacktestConfig) response.Response {
//nolint:staticcheck // not yet migrated to OpenFeature
if !srv.featureManager.IsEnabled(c.Req.Context(), featuremgmt.FlagAlertingBacktesting) {
return ErrResp(http.StatusNotFound, nil, "Backgtesting API is not enabled")
}
@@ -6,6 +6,7 @@ import (
"encoding/json"
"errors"
"fmt"
"maps"
"net/url"
"slices"
"sort"
@@ -26,6 +27,9 @@ import (
ngmodels "github.com/grafana/grafana/pkg/services/ngalert/models"
"github.com/grafana/grafana/pkg/services/ngalert/state"
"github.com/grafana/grafana/pkg/util"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
)
type RuleStoreReader interface {
@@ -54,6 +58,9 @@ type PrometheusSrv struct {
provenanceStore ProvenanceStore
}
// Package-level OpenTelemetry tracer per Grafana instrumentation conventions.
var tracer = otel.Tracer("github.com/grafana/grafana/pkg/services/ngalert/api/prometheus")
func NewPrometheusSrv(log log.Logger, manager state.AlertInstanceManager, status StatusReader, store RuleStoreReader, authz RuleGroupAccessControlService, provenanceStore ProvenanceStore) *PrometheusSrv {
return &PrometheusSrv{
log,
@@ -219,6 +226,14 @@ func GetStatesFromQuery(v url.Values) (map[eval.State]struct{}, error) {
return states, nil
}
func MapStateSetToStrings(stateSet map[eval.State]struct{}) []string {
states := make([]string, 0, len(stateSet))
for state := range stateSet {
states = append(states, state.String())
}
return states
}
func GetHealthFromQuery(v url.Values) (map[string]struct{}, error) {
health := make(map[string]struct{})
for _, s := range v["health"] {
@@ -252,6 +267,13 @@ func (srv PrometheusSrv) RouteGetRuleStatuses(c *contextmodel.ReqContext) respon
// As we are using req.Form directly, this triggers a call to ParseForm() if needed.
c.Query("")
ctx, span := tracer.Start(c.Req.Context(), "api.prometheus.RouteGetRuleStatuses")
defer span.End()
// Propagate the new context so child spans can attach to it.
c.Req = c.Req.WithContext(ctx)
orgID := c.GetOrgID()
span.SetAttributes(attribute.Int64("org_id", orgID))
ruleResponse := apimodels.RuleResponse{
DiscoveryBase: apimodels.DiscoveryBase{
Status: "success",
@@ -261,13 +283,14 @@ func (srv PrometheusSrv) RouteGetRuleStatuses(c *contextmodel.ReqContext) respon
},
}
namespaceMap, err := srv.store.GetUserVisibleNamespaces(c.Req.Context(), c.GetOrgID(), c.SignedInUser)
namespaceMap, err := srv.store.GetUserVisibleNamespaces(c.Req.Context(), orgID, c.SignedInUser)
if err != nil {
ruleResponse.Status = "error"
ruleResponse.Error = fmt.Sprintf("failed to get namespaces visible to the user: %s", err.Error())
ruleResponse.ErrorType = apiv1.ErrServer
return response.JSON(ruleResponse.HTTPStatusCode(), ruleResponse)
}
span.AddEvent("User visible namespaces retrieved")
allowedNamespaces := map[string]string{}
for namespaceUID, folder := range namespaceMap {
@@ -283,6 +306,8 @@ func (srv PrometheusSrv) RouteGetRuleStatuses(c *contextmodel.ReqContext) respon
allowedNamespaces[namespaceUID] = folder.Fullpath
}
}
span.AddEvent("User permissions checked")
span.SetAttributes(attribute.Int("allowedNamespaces", len(allowedNamespaces)))
provenanceRecords, err := srv.provenanceStore.GetProvenances(c.Req.Context(), c.GetOrgID(), (&ngmodels.AlertRule{}).ResourceType())
if err != nil {
@@ -297,7 +322,7 @@ func (srv PrometheusSrv) RouteGetRuleStatuses(c *contextmodel.ReqContext) respon
srv.store,
RuleGroupStatusesOptions{
Ctx: c.Req.Context(),
OrgID: c.OrgID,
OrgID: orgID,
Query: c.Req.Form,
AllowedNamespaces: allowedNamespaces,
},
@@ -405,6 +430,10 @@ func RuleAlertStateMutatorGenerator(manager state.AlertInstanceManager) RuleAler
}
func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opts RuleGroupStatusesOptions, ruleStatusMutator RuleStatusMutator, alertStateMutator RuleAlertStateMutator, provenanceRecords map[string]ngmodels.Provenance) apimodels.RuleResponse {
ctx, span := tracer.Start(opts.Ctx, "api.prometheus.PrepareRuleGroupStatusesV2")
defer span.End()
opts.Ctx = ctx
ruleResponse := apimodels.RuleResponse{
DiscoveryBase: apimodels.DiscoveryBase{
Status: "success",
@@ -428,9 +457,17 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
ruleResponse.ErrorType = apiv1.ErrBadData
return ruleResponse
}
span.SetAttributes(
attribute.String("dashboard_uid", dashboardUID),
attribute.Int64("panel_id", panelID),
)
limitRulesPerGroup := getInt64WithDefault(opts.Query, "limit_rules", -1)
limitAlertsPerRule := getInt64WithDefault(opts.Query, "limit_alerts", -1)
span.SetAttributes(
attribute.Int64("limit_rules", limitRulesPerGroup),
attribute.Int64("limit_alerts", limitAlertsPerRule),
)
matchers, err := getMatchersFromQuery(opts.Query)
if err != nil {
ruleResponse.Status = "error"
@@ -438,6 +475,8 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
ruleResponse.ErrorType = apiv1.ErrBadData
return ruleResponse
}
span.SetAttributes(attribute.Int("matcher_count", len(matchers)))
stateFilterSet, err := GetStatesFromQuery(opts.Query)
if err != nil {
ruleResponse.Status = "error"
@@ -445,6 +484,10 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
ruleResponse.ErrorType = apiv1.ErrBadData
return ruleResponse
}
span.SetAttributes(
attribute.Int("state_filter_count", len(stateFilterSet)),
attribute.StringSlice("state_filter", MapStateSetToStrings(stateFilterSet)),
)
healthFilterSet, err := GetHealthFromQuery(opts.Query)
if err != nil {
@@ -453,11 +496,18 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
ruleResponse.ErrorType = apiv1.ErrBadData
return ruleResponse
}
span.SetAttributes(
attribute.Int("health_filter_count", len(healthFilterSet)),
attribute.StringSlice("health_filter", slices.Collect(maps.Keys(healthFilterSet))),
)
var labelOptions []ngmodels.LabelOption
if !getBoolWithDefault(opts.Query, queryIncludeInternalLabels, false) {
labelOptions = append(labelOptions, ngmodels.WithoutInternalLabels())
}
span.SetAttributes(
attribute.Bool("include_internal_labels", len(labelOptions) == 0),
)
if len(opts.AllowedNamespaces) == 0 {
log.Debug("User does not have access to any namespaces")
@@ -476,19 +526,36 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
}
}
span.SetAttributes(
attribute.Bool("folder_uid_set", folderUID != ""),
attribute.Int("namespace_count", len(namespaceUIDs)),
)
ruleGroups := opts.Query["rule_group"]
ruleUIDs := opts.Query["rule_uid"]
span.SetAttributes(
attribute.Int("rule_group_count", len(ruleGroups)),
attribute.Int("rule_uid_count", len(ruleUIDs)),
)
receiverName := opts.Query.Get("receiver_name")
span.SetAttributes(attribute.Bool("receiver_name_set", receiverName != ""))
title := opts.Query.Get("search.rule_name")
span.SetAttributes(attribute.Bool("search_rule_name_set", title != ""))
searchRuleGroup := opts.Query.Get("search.rule_group")
span.SetAttributes(attribute.Bool("search_rule_group_set", searchRuleGroup != ""))
var ruleType ngmodels.RuleTypeFilter
switch ngmodels.RuleType(opts.Query.Get("rule_type")) {
case ngmodels.RuleTypeAlerting:
ruleType = ngmodels.RuleTypeFilterAlerting
span.SetAttributes(attribute.Bool("alerting_only", true))
case ngmodels.RuleTypeRecording:
ruleType = ngmodels.RuleTypeFilterRecording
span.SetAttributes(attribute.Bool("recording_only", true))
default:
ruleType = ngmodels.RuleTypeFilterAll
}
@@ -507,11 +574,23 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
maxGroups := getInt64WithDefault(opts.Query, "group_limit", -1)
maxRules := getInt64WithDefault(opts.Query, "rule_limit", -1)
nextToken := opts.Query.Get("group_next_token")
span.SetAttributes(
attribute.Int64("group_limit", maxGroups),
attribute.Int64("rule_limit", maxRules),
attribute.Bool("group_next_token_set", nextToken != ""),
)
if maxGroups == 0 || maxRules == 0 {
return ruleResponse
}
ruleNames := opts.Query["rule_name"]
ruleNamesSet := make(map[string]struct{}, len(ruleNames))
for _, rn := range ruleNames {
ruleNamesSet[rn] = struct{}{}
}
span.SetAttributes(attribute.Int("rule_name_count", len(ruleNamesSet)))
byGroupQuery := ngmodels.ListAlertRulesExtendedQuery{
ListAlertRulesQuery: ngmodels.ListAlertRulesQuery{
OrgID: opts.OrgID,
@@ -536,12 +615,11 @@ func PrepareRuleGroupStatusesV2(log log.Logger, store ListAlertRulesStoreV2, opt
ruleResponse.ErrorType = apiv1.ErrServer
return ruleResponse
}
ruleNames := opts.Query["rule_name"]
ruleNamesSet := make(map[string]struct{}, len(ruleNames))
for _, rn := range ruleNames {
ruleNamesSet[rn] = struct{}{}
}
span.SetAttributes(
attribute.Int("store_rule_list_len", len(ruleList)),
attribute.Bool("store_continue_token_set", continueToken != ""),
)
span.AddEvent("Alert rules retrieved from store")
groupedRules := getGroupedRules(log, ruleList, ruleNamesSet, opts.AllowedNamespaces)
rulesTotals := make(map[string]int64, len(groupedRules))
+4
View File
@@ -194,8 +194,11 @@ func (ng *AlertNG) init() error {
var opts []notifier.Option
moaLogger := log.New("ngalert.multiorg.alertmanager")
crypto := notifier.NewCrypto(ng.SecretsService, ng.store, moaLogger)
//nolint:staticcheck // not yet migrated to OpenFeature
remotePrimary := ng.FeatureToggles.IsEnabled(initCtx, featuremgmt.FlagAlertmanagerRemotePrimary)
//nolint:staticcheck // not yet migrated to OpenFeature
remoteSecondary := ng.FeatureToggles.IsEnabled(initCtx, featuremgmt.FlagAlertmanagerRemoteSecondary)
//nolint:staticcheck // not yet migrated to OpenFeature
remoteSecondaryWithRemoteState := ng.FeatureToggles.IsEnabled(initCtx, featuremgmt.FlagAlertmanagerRemoteSecondaryWithRemoteState)
if remotePrimary || remoteSecondary || remoteSecondaryWithRemoteState {
m := ng.Metrics.GetRemoteAlertmanagerMetrics()
@@ -717,6 +720,7 @@ func configureNotificationHistorian(
l log.Logger,
tracer tracing.Tracer,
) (nfstatus.NotificationHistorian, error) {
//nolint:staticcheck // not yet migrated to OpenFeature
if !featureToggles.IsEnabled(ctx, featuremgmt.FlagAlertingNotificationHistory) || !cfg.Enabled {
met.Info.Set(0)
return nil, nil
+1
View File
@@ -89,6 +89,7 @@ func (d *AlertsRouter) SyncAndApplyConfigFromDatabase(ctx context.Context) error
d.logger.Debug("Attempting to sync admin configs", "count", len(cfgs))
//nolint:staticcheck // not yet migrated to OpenFeature
disableExternal := d.featureManager.IsEnabled(ctx, featuremgmt.FlagAlertingDisableSendAlertsExternal)
orgsFound := make(map[int64]struct{}, len(cfgs))
+1
View File
@@ -1097,6 +1097,7 @@ func (st DBstore) GetAlertRulesForScheduling(ctx context.Context, query *ngmodel
continue
}
}
//nolint:staticcheck // not yet migrated to OpenFeature
if st.FeatureToggles.IsEnabled(ctx, featuremgmt.FlagAlertingQueryOptimization) {
if optimizations, err := OptimizeAlertQueries(converted.Data); err != nil {
st.Logger.Error("Could not migrate rule from range to instant query", "rule", rule.UID, "err", err)
@@ -87,10 +87,12 @@ func ProvideSyncer(
}
func (s *syncer) Sync(ctx context.Context, source install.Source, installedPlugins []*plugins.Plugin) error {
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.featureToggles.IsEnabled(ctx, featuremgmt.FlagPluginInstallAPISync) {
return nil
}
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.featureToggles.IsEnabled(ctx, featuremgmt.FlagPluginStoreServiceLoading) {
logging.DefaultLogger.Warn("pluginInstallAPISync is enabled, but pluginStoreServiceLoading is disabled. skipping plugin sync.")
return nil
@@ -161,7 +161,7 @@ func TestIntegrationPluginManager(t *testing.T) {
pr := prometheus.ProvideService(hcp)
tmpo := tempo.ProvideService(hcp, tracer)
td := testdatasource.ProvideService()
pg := postgres.ProvideService(cfg)
pg := postgres.ProvideService()
my := mysql.ProvideService()
ms := mssql.ProvideService(cfg)
db := db.InitTestDB(t, sqlstore.InitTestDBOpt{Cfg: cfg})
+2
View File
@@ -61,6 +61,7 @@ func UpdatePreferencesFor(ctx context.Context,
Navbar: dtoCmd.Navbar,
}
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(ctx, featuremgmt.FlagLocaleFormatPreference) {
saveCmd.RegionalFormat = dtoCmd.RegionalFormat
}
@@ -101,6 +102,7 @@ func GetPreferencesFor(ctx context.Context,
dto.Language = &preference.JSONData.Language
}
//nolint:staticcheck // not yet migrated to OpenFeature
if features.IsEnabled(ctx, featuremgmt.FlagLocaleFormatPreference) {
if preference.JSONData.RegionalFormat != "" {
dto.RegionalFormat = &preference.JSONData.RegionalFormat
@@ -43,6 +43,7 @@ func ProvidePromTypeMigrationProvider(
}
func (s *PromTypeMigrationProviderImpl) Run(ctx context.Context) error {
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.features.IsEnabled(ctx, featuremgmt.FlagPrometheusTypeMigration) {
return nil
}
+1
View File
@@ -37,6 +37,7 @@ func (rs *RenderingService) GetRenderUser(ctx context.Context, key string) (*Ren
var renderUser *RenderUser
//nolint:staticcheck // not yet migrated to OpenFeature
if looksLikeJWT(key) && rs.features.IsEnabled(ctx, featuremgmt.FlagRenderAuthJWT) {
from = "jwt"
renderUser = rs.getRenderUserFromJWT(key)
+4
View File
@@ -122,6 +122,7 @@ func (s *SecretsService) registerUsageMetrics() {
// Enabled / disabled
usageMetrics["stats.encryption.envelope_encryption_enabled.count"] = 0
//nolint:staticcheck // not yet migrated to OpenFeature
if !s.features.IsEnabled(ctx, featuremgmt.FlagDisableEnvelopeEncryption) {
usageMetrics["stats.encryption.envelope_encryption_enabled.count"] = 1
}
@@ -167,6 +168,7 @@ func (s *SecretsService) Encrypt(ctx context.Context, payload []byte, opt secret
defer span.End()
// Use legacy encryption service if featuremgmt.FlagDisableEnvelopeEncryption toggle is on
//nolint:staticcheck // not yet migrated to OpenFeature
if s.features.IsEnabled(ctx, featuremgmt.FlagDisableEnvelopeEncryption) {
return s.enc.Encrypt(ctx, payload, s.cfg.SecretKey)
}
@@ -343,6 +345,7 @@ func (s *SecretsService) Decrypt(ctx context.Context, payload []byte) ([]byte, e
// If encrypted with envelope encryption, the feature is disabled and
// no provider is initialized, then we throw an error.
//nolint:staticcheck // not yet migrated to OpenFeature
if s.encryptedWithEnvelopeEncryption(payload) &&
s.features.IsEnabled(ctx, featuremgmt.FlagDisableEnvelopeEncryption) &&
!s.providersInitialized() {
@@ -480,6 +483,7 @@ func (s *SecretsService) RotateDataKeys(ctx context.Context) error {
func (s *SecretsService) ReEncryptDataKeys(ctx context.Context) error {
s.log.Info("Data keys re-encryption triggered")
//nolint:staticcheck // not yet migrated to OpenFeature
if s.features.IsEnabled(ctx, featuremgmt.FlagDisableEnvelopeEncryption) {
s.log.Info("Envelope encryption is not enabled but trying to init providers anyway...")
+22
View File
@@ -13,6 +13,8 @@ service ResourceIndex {
// Get the resource stats
rpc GetStats(ResourceStatsRequest) returns (ResourceStatsResponse);
rpc RebuildIndexes(RebuildIndexesRequest) returns (RebuildIndexesResponse);
}
// Get statistics across multiple resources
@@ -138,3 +140,23 @@ message ResourceSearchResponse {
// Facet results
map<string,Facet> facet = 7;
}
message RebuildIndexesRequest {
// Namespace (tenant) must be the same as all keys' namespace
string namespace = 1;
// List of ResourceKeys (Namespace + Group + Resource)
repeated ResourceKey keys = 2;
}
message RebuildIndexesResponse {
// Total count of rebuilt indexes
int64 rebuildCount = 1;
// Result message
string details = 2;
// Error details
ErrorResult error = 3;
}
+75 -1
View File
@@ -1,4 +1,4 @@
// Code generated by mockery v2.53.4. DO NOT EDIT.
// Code generated by mockery v2.53.5. DO NOT EDIT.
package resource
@@ -838,6 +838,80 @@ func (_c *MockResourceClient_Read_Call) RunAndReturn(run func(context.Context, *
return _c
}
// RebuildIndexes provides a mock function with given fields: ctx, in, opts
func (_m *MockResourceClient) RebuildIndexes(ctx context.Context, in *resourcepb.RebuildIndexesRequest, opts ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error) {
_va := make([]interface{}, len(opts))
for _i := range opts {
_va[_i] = opts[_i]
}
var _ca []interface{}
_ca = append(_ca, ctx, in)
_ca = append(_ca, _va...)
ret := _m.Called(_ca...)
if len(ret) == 0 {
panic("no return value specified for RebuildIndexes")
}
var r0 *resourcepb.RebuildIndexesResponse
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, *resourcepb.RebuildIndexesRequest, ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error)); ok {
return rf(ctx, in, opts...)
}
if rf, ok := ret.Get(0).(func(context.Context, *resourcepb.RebuildIndexesRequest, ...grpc.CallOption) *resourcepb.RebuildIndexesResponse); ok {
r0 = rf(ctx, in, opts...)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*resourcepb.RebuildIndexesResponse)
}
}
if rf, ok := ret.Get(1).(func(context.Context, *resourcepb.RebuildIndexesRequest, ...grpc.CallOption) error); ok {
r1 = rf(ctx, in, opts...)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// MockResourceClient_RebuildIndexes_Call is a *mock.Call that shadows Run/Return methods with type explicit version for method 'RebuildIndexes'
type MockResourceClient_RebuildIndexes_Call struct {
*mock.Call
}
// RebuildIndexes is a helper method to define mock.On call
// - ctx context.Context
// - in *resourcepb.RebuildIndexesRequest
// - opts ...grpc.CallOption
func (_e *MockResourceClient_Expecter) RebuildIndexes(ctx interface{}, in interface{}, opts ...interface{}) *MockResourceClient_RebuildIndexes_Call {
return &MockResourceClient_RebuildIndexes_Call{Call: _e.mock.On("RebuildIndexes",
append([]interface{}{ctx, in}, opts...)...)}
}
func (_c *MockResourceClient_RebuildIndexes_Call) Run(run func(ctx context.Context, in *resourcepb.RebuildIndexesRequest, opts ...grpc.CallOption)) *MockResourceClient_RebuildIndexes_Call {
_c.Call.Run(func(args mock.Arguments) {
variadicArgs := make([]grpc.CallOption, len(args)-2)
for i, a := range args[2:] {
if a != nil {
variadicArgs[i] = a.(grpc.CallOption)
}
}
run(args[0].(context.Context), args[1].(*resourcepb.RebuildIndexesRequest), variadicArgs...)
})
return _c
}
func (_c *MockResourceClient_RebuildIndexes_Call) Return(_a0 *resourcepb.RebuildIndexesResponse, _a1 error) *MockResourceClient_RebuildIndexes_Call {
_c.Call.Return(_a0, _a1)
return _c
}
func (_c *MockResourceClient_RebuildIndexes_Call) RunAndReturn(run func(context.Context, *resourcepb.RebuildIndexesRequest, ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error)) *MockResourceClient_RebuildIndexes_Call {
_c.Call.Return(run)
return _c
}
// Search provides a mock function with given fields: ctx, in, opts
func (_m *MockResourceClient) Search(ctx context.Context, in *resourcepb.ResourceSearchRequest, opts ...grpc.CallOption) (*resourcepb.ResourceSearchResponse, error) {
_va := make([]interface{}, len(opts))
+86 -9
View File
@@ -234,6 +234,9 @@ func combineRebuildRequests(a, b rebuildRequest) (c rebuildRequest, ok bool) {
ret.lastImportTime = b.lastImportTime
}
// Combine complete channels
ret.completeChannels = append(a.completeChannels, b.completeChannels...)
return ret, true
}
@@ -512,6 +515,52 @@ func (s *searchSupport) GetStats(ctx context.Context, req *resourcepb.ResourceSt
return rsp, nil
}
func (s *searchSupport) RebuildIndexes(ctx context.Context, req *resourcepb.RebuildIndexesRequest) (*resourcepb.RebuildIndexesResponse, error) {
ctx, span := tracer.Start(ctx, "resource.searchSupport.RebuildIndexes")
defer span.End()
filterKeys := make([]NamespacedResource, len(req.Keys))
for _, key := range req.Keys {
if req.Namespace != key.Namespace {
return &resourcepb.RebuildIndexesResponse{
Error: NewBadRequestError("key namespace does not match request namespace"),
}, nil
}
filterKeys = append(filterKeys, NamespacedResource{
Namespace: key.Namespace,
Group: key.Group,
Resource: key.Resource,
})
}
importTimes, err := s.getLastImportTimes(ctx)
if err != nil {
return &resourcepb.RebuildIndexesResponse{
Error: AsErrorResult(err),
}, nil
}
completeChs := s.findIndexesToRebuild(importTimes, filterKeys, time.Now())
rebuildCount := len(completeChs)
for _, ch := range completeChs {
select {
case <-ch:
continue
case <-ctx.Done(): // request was done before all indexes rebuilt
return &resourcepb.RebuildIndexesResponse{
RebuildCount: int64(rebuildCount),
Details: fmt.Sprintf("returning before all index rebuilds completed for %d indexes", rebuildCount),
}, nil
}
}
// All rebuilds completed successfully
return &resourcepb.RebuildIndexesResponse{
RebuildCount: int64(rebuildCount),
Details: fmt.Sprintf("completed %d index rebuilds", rebuildCount),
}, nil
}
func (s *searchSupport) buildIndexes(ctx context.Context) (int, error) {
totalBatchesIndexed := 0
group := errgroup.Group{}
@@ -602,16 +651,23 @@ func (s *searchSupport) runPeriodicScanForIndexesToRebuild(ctx context.Context)
if err != nil {
s.log.Error("failed to get import times", "error", err)
}
s.findIndexesToRebuild(importTimes, time.Now())
s.findIndexesToRebuild(importTimes, nil, time.Now())
}
}
}
func (s *searchSupport) findIndexesToRebuild(lastImportTimes map[NamespacedResource]time.Time, now time.Time) {
func (s *searchSupport) findIndexesToRebuild(lastImportTimes map[NamespacedResource]time.Time, filterKeys []NamespacedResource, now time.Time) []chan struct{} {
// Check all open indexes and see if any of them need to be rebuilt.
// This is done periodically to make sure that the indexes are up to date.
keys := s.search.GetOpenIndexes()
var keys []NamespacedResource
if filterKeys != nil {
keys = filterKeys
} else {
keys = s.search.GetOpenIndexes()
}
var completeChs []chan struct{}
for _, key := range keys {
idx := s.search.GetIndex(key)
if idx == nil {
@@ -638,18 +694,17 @@ func (s *searchSupport) findIndexesToRebuild(lastImportTimes map[NamespacedResou
}
if shouldRebuildIndex(bi, s.minBuildVersion, minBuildTime, lastImportTime, nil) {
s.rebuildQueue.Add(rebuildRequest{
NamespacedResource: key,
minBuildTime: minBuildTime,
minBuildVersion: s.minBuildVersion,
lastImportTime: lastImportTime,
})
completeCh := make(chan struct{})
completeChs = append(completeChs, completeCh)
rebuildReq := newRebuildRequest(key, minBuildTime, lastImportTime, s.minBuildVersion, completeCh)
s.rebuildQueue.Add(rebuildReq)
if s.indexMetrics != nil {
s.indexMetrics.RebuildQueueLength.Set(float64(s.rebuildQueue.Len()))
}
}
}
return completeChs
}
func (s *searchSupport) getLastImportTimes(ctx context.Context) (map[NamespacedResource]time.Time, error) {
@@ -690,6 +745,12 @@ func (s *searchSupport) rebuildIndex(ctx context.Context, req rebuildRequest) {
l := s.log.New("namespace", req.Namespace, "group", req.Group, "resource", req.Resource)
defer func() {
for _, ch := range req.completeChannels {
close(ch)
}
}()
idx := s.search.GetIndex(req.NamespacedResource)
if idx == nil {
span.AddEvent("index not found")
@@ -782,6 +843,22 @@ type rebuildRequest struct {
minBuildTime time.Time // if not zero, rebuild index if it has been built before this timestamp
lastImportTime time.Time // if not zero, rebuild index if it has been built before this timestamp.
minBuildVersion *semver.Version // if not nil, rebuild index with build version older than this.
completeChannels []chan<- struct{} // signal rebuild index is complete
}
func newRebuildRequest(key NamespacedResource, minBuildTime, lastImportTime time.Time, minBuildVersion *semver.Version, completeCh chan<- struct{}) rebuildRequest {
var completeChannels []chan<- struct{} // setup a list as requests can be combined
if completeCh != nil {
completeChannels = []chan<- struct{}{completeCh}
}
return rebuildRequest{
NamespacedResource: key,
minBuildTime: minBuildTime,
minBuildVersion: minBuildVersion,
lastImportTime: lastImportTime,
completeChannels: completeChannels,
}
}
func (s *searchSupport) getOrCreateIndex(ctx context.Context, stats *SearchStats, key NamespacedResource, reason string) (ResourceIndex, error) {
@@ -211,6 +211,11 @@ func (s *searchWrapper) Search(ctx context.Context, in *resourcepb.ResourceSearc
return client.Search(ctx, in, opts...)
}
func (s *searchWrapper) RebuildIndexes(ctx context.Context, in *resourcepb.RebuildIndexesRequest,
opts ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error) {
return s.unifiedClient.RebuildIndexes(ctx, in, opts...)
}
// compareSearchResults compares legacy and unified search results and logs/metrics the outcome
func (s *searchWrapper) compareSearchResults(legacyResponse, unifiedResponse *resourcepb.ResourceSearchResponse, requestKey *resourcepb.ResourceKey) {
if legacyResponse == nil || unifiedResponse == nil {
@@ -112,6 +112,11 @@ func (m *MockResourceIndexClient) GetStats(ctx context.Context, in *resourcepb.R
return args.Get(0).(*resourcepb.ResourceStatsResponse), args.Error(1)
}
func (m *MockResourceIndexClient) RebuildIndexes(ctx context.Context, in *resourcepb.RebuildIndexesRequest, opts ...grpc.CallOption) (*resourcepb.RebuildIndexesResponse, error) {
args := m.Called(ctx, in, opts)
return args.Get(0).(*resourcepb.RebuildIndexesResponse), args.Error(1)
}
func setupTestSearchClient(t *testing.T) (schema.GroupResource, *MockResourceIndexClient, *MockResourceIndexClient, featuremgmt.FeatureToggles) {
t.Helper()
gr := schema.GroupResource{Group: "test", Resource: "items"}
@@ -2,8 +2,12 @@ package resource
import (
"context"
"errors"
"fmt"
"hash/fnv"
"math/rand"
"sync"
"sync/atomic"
"time"
"github.com/grafana/dskit/ring"
@@ -114,6 +118,100 @@ func (ds *distributorServer) GetStats(ctx context.Context, r *resourcepb.Resourc
return client.GetStats(ctx, r)
}
func (ds *distributorServer) RebuildIndexes(ctx context.Context, r *resourcepb.RebuildIndexesRequest) (*resourcepb.RebuildIndexesResponse, error) {
ctx, span := ds.tracing.Start(ctx, "distributor.RebuildIndexes")
defer span.End()
// validate input
for _, key := range r.Keys {
if r.Namespace != key.Namespace {
return &resourcepb.RebuildIndexesResponse{
Error: NewBadRequestError("key namespace does not match request namespace"),
}, nil
}
}
// distribute the request to all search pods to minimize risk of stale index
// it will not rebuild on those which don't have the index open
rs, err := ds.ring.GetAllHealthy(searchRingRead)
if err != nil {
return nil, fmt.Errorf("failed to get all healthy instances from the ring")
}
err = grpc.SetHeader(ctx, metadata.Pairs("proxied-instance-id", "all"))
if err != nil {
ds.log.Debug("error setting grpc header", "err", err)
}
md, ok := metadata.FromIncomingContext(ctx)
if !ok {
md = make(metadata.MD)
}
rCtx := userutils.InjectOrgID(metadata.NewOutgoingContext(ctx, md), r.Namespace)
var wg sync.WaitGroup
var totalRebuildCount atomic.Int64
detailsCh := make(chan string, len(rs.Instances))
errorCh := make(chan error, len(rs.Instances))
for _, inst := range rs.Instances {
wg.Add(1)
go func() {
defer wg.Done()
client, err := ds.clientPool.GetClientForInstance(inst)
if err != nil {
errorCh <- fmt.Errorf("instance %s: failed to get client, %w", inst.Id, err)
return
}
rsp, err := client.(*RingClient).Client.RebuildIndexes(rCtx, r)
if err != nil {
errorCh <- fmt.Errorf("instance %s: failed to distribute rebuild index request, %w", inst.Id, err)
return
}
if rsp.Error != nil {
errorCh <- fmt.Errorf("instance %s: rebuild index request returned the error %s", inst.Id, rsp.Error.Message)
return
}
if rsp.Details != "" {
detailsCh <- fmt.Sprintf("{instance: %s, details: %s}", inst.Id, rsp.Details)
}
totalRebuildCount.Add(rsp.RebuildCount)
}()
}
wg.Wait()
close(errorCh)
close(detailsCh)
errs := make([]error, 0, len(errorCh))
for err := range errorCh {
ds.log.Error("rebuild indexes call failed with %w", err)
errs = append(errs, err)
}
var details string
for d := range detailsCh {
if len(details) > 0 {
details += ", "
}
details += d
}
response := &resourcepb.RebuildIndexesResponse{
RebuildCount: totalRebuildCount.Load(),
Details: details,
}
if len(errs) > 0 {
response.Error = AsErrorResult(errors.Join(errs...))
}
return response, nil
}
func (ds *distributorServer) CountManagedObjects(ctx context.Context, r *resourcepb.CountManagedObjectsRequest) (*resourcepb.CountManagedObjectsResponse, error) {
ctx, span := ds.tracing.Start(ctx, "distributor.CountManagedObjects")
defer span.End()
+94 -6
View File
@@ -10,6 +10,8 @@ import (
"time"
"github.com/Masterminds/semver"
"github.com/google/go-cmp/cmp"
"github.com/google/go-cmp/cmp/cmpopts"
"github.com/grafana/authlib/types"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
@@ -85,7 +87,8 @@ func (m *MockDocumentBuilder) BuildDocument(ctx context.Context, key *resourcepb
// mockStorageBackend implements StorageBackend for testing
type mockStorageBackend struct {
resourceStats []ResourceStats
resourceStats []ResourceStats
lastImportTimes []ResourceLastImportTime
}
func (m *mockStorageBackend) GetResourceStats(ctx context.Context, nsr NamespacedResource, minCount int) ([]ResourceStats, error) {
@@ -127,7 +130,11 @@ func (m *mockStorageBackend) ListModifiedSince(ctx context.Context, key Namespac
func (m *mockStorageBackend) GetResourceLastImportTimes(ctx context.Context) iter.Seq2[ResourceLastImportTime, error] {
return func(yield func(ResourceLastImportTime, error) bool) {
yield(ResourceLastImportTime{}, errors.New("not implemented"))
for _, ti := range m.lastImportTimes {
if !yield(ti, nil) {
return
}
}
}
}
@@ -605,14 +612,14 @@ func TestFindIndexesForRebuild(t *testing.T) {
{Namespace: "resource-v6", Group: "group", Resource: dashboardv1.DASHBOARD_RESOURCE}: lastImportTime,
}
support.findIndexesToRebuild(importTimes, now)
support.findIndexesToRebuild(importTimes, nil, now)
require.Equal(t, 7, support.rebuildQueue.Len())
now5m := now.Add(5 * time.Minute)
// Running findIndexesToRebuild again should not add any new indexes to the rebuild queue, and all existing
// ones should be "combined" with new ones (this will "bump" minBuildTime)
support.findIndexesToRebuild(importTimes, now5m)
support.findIndexesToRebuild(importTimes, nil, now5m)
require.Equal(t, 7, support.rebuildQueue.Len())
// Values that we expect to find in rebuild requests.
@@ -621,7 +628,7 @@ func TestFindIndexesForRebuild(t *testing.T) {
minBuildTimeDashboard := now5m.Add(-1 * time.Hour)
vals := support.rebuildQueue.Elements()
require.ElementsMatch(t, vals, []rebuildRequest{
expected := []rebuildRequest{
{NamespacedResource: NamespacedResource{Namespace: "resource-2h-v5", Group: "group", Resource: "folder"}, minBuildVersion: minBuildVersion, minBuildTime: minBuildTime},
{NamespacedResource: NamespacedResource{Namespace: "resource-10h-v5", Group: "group", Resource: "folder"}, minBuildVersion: minBuildVersion, minBuildTime: minBuildTime},
{NamespacedResource: NamespacedResource{Namespace: "resource-10h-v6", Group: "group", Resource: "folder"}, minBuildVersion: minBuildVersion, minBuildTime: minBuildTime},
@@ -631,7 +638,10 @@ func TestFindIndexesForRebuild(t *testing.T) {
{NamespacedResource: NamespacedResource{Namespace: "resource-2h-v6", Group: "group", Resource: dashboardv1.DASHBOARD_RESOURCE}, minBuildVersion: minBuildVersion, minBuildTime: minBuildTimeDashboard},
{NamespacedResource: NamespacedResource{Namespace: "resource-recently-imported", Group: "group", Resource: dashboardv1.DASHBOARD_RESOURCE}, minBuildVersion: minBuildVersion, minBuildTime: minBuildTimeDashboard, lastImportTime: lastImportTime},
})
}
if diff := cmp.Diff(expected, vals, cmpopts.IgnoreFields(rebuildRequest{}, "completeChannels"), cmp.AllowUnexported(rebuildRequest{})); diff != "" {
t.Errorf("rebuildQueue mismatch (-want +got):\n%s", diff)
}
}
func TestRebuildIndexes(t *testing.T) {
@@ -748,3 +758,81 @@ func checkRebuildIndex(t *testing.T, support *searchSupport, req rebuildRequest,
require.Nil(t, idxAfter, "index should not exist after rebuildIndex")
}
}
func TestRebuildIndexesForResource(t *testing.T) {
key := NamespacedResource{Namespace: "ns", Group: "group", Resource: "resource"}
storage := &mockStorageBackend{
resourceStats: []ResourceStats{
{NamespacedResource: key, Count: 50, ResourceVersion: 11111111},
},
lastImportTimes: []ResourceLastImportTime{{
NamespacedResource: key,
LastImportTime: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC),
}},
}
search := &mockSearchBackend{}
supplier := &TestDocumentBuilderSupplier{
GroupsResources: map[string]string{
"group": "resource",
},
}
opts := SearchOptions{
Backend: search,
Resources: supplier,
InitMinCount: 1,
}
support, err := newSearchSupport(opts, storage, nil, nil, nil, nil)
require.NoError(t, err)
require.NotNil(t, support)
err = support.init(t.Context())
require.NoError(t, err)
require.Equal(t, 0, support.rebuildQueue.Len())
// invalid request
rebuildReq := &resourcepb.RebuildIndexesRequest{
Namespace: "some-other-namespace",
Keys: []*resourcepb.ResourceKey{{
Namespace: key.Namespace,
Group: key.Group,
Resource: key.Resource,
}}}
rsp, err := support.RebuildIndexes(t.Context(), rebuildReq)
require.NoError(t, err)
require.Equal(t, "key namespace does not match request namespace", rsp.Error.Message)
rebuildReq.Namespace = key.Namespace
// cached index info
search.cache[key] = &MockResourceIndex{
buildInfo: IndexBuildInfo{BuildVersion: semver.MustParse("5.0.0"), BuildTime: time.Date(2025, 1, 1, 12, 0, 0, 0, time.UTC)},
}
// old import time will not be rebuilt
storage.lastImportTimes = []ResourceLastImportTime{{
NamespacedResource: key,
LastImportTime: time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC),
}}
rsp, err = support.RebuildIndexes(t.Context(), rebuildReq)
require.NoError(t, err)
require.Equal(t, int64(0), rsp.RebuildCount)
require.Equal(t, 0, support.rebuildQueue.Len())
// recent import time gets added to rebuild queue and processed
storage.lastImportTimes = []ResourceLastImportTime{{
NamespacedResource: key,
LastImportTime: time.Date(2026, 1, 1, 12, 0, 0, 0, time.UTC),
}}
rsp, err = support.RebuildIndexes(t.Context(), rebuildReq)
require.NoError(t, err)
require.Equal(t, int64(1), rsp.RebuildCount)
// rebuild waited for rebuild queue to process
require.Equal(t, 0, support.rebuildQueue.Len())
}
+8
View File
@@ -1541,3 +1541,11 @@ func (s *server) runInQueue(ctx context.Context, tenantID string, runnable func(
return queueCtx.Err() // Timed out or canceled while waiting for execution.
}
}
func (s *server) RebuildIndexes(ctx context.Context, req *resourcepb.RebuildIndexesRequest) (*resourcepb.RebuildIndexesResponse, error) {
if s.search == nil {
return nil, fmt.Errorf("search index not configured")
}
return s.search.RebuildIndexes(ctx, req)
}
+204 -61
View File
@@ -386,6 +386,123 @@ func (x *ResourceSearchResponse) GetFacet() map[string]*ResourceSearchResponse_F
return nil
}
type RebuildIndexesRequest struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Namespace (tenant)
Namespace string `protobuf:"bytes,1,opt,name=namespace,proto3" json:"namespace,omitempty"`
// List of ResourceKeys (Namespace + Group + Resource)
Keys []*ResourceKey `protobuf:"bytes,2,rep,name=keys,proto3" json:"keys,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *RebuildIndexesRequest) Reset() {
*x = RebuildIndexesRequest{}
mi := &file_search_proto_msgTypes[4]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *RebuildIndexesRequest) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*RebuildIndexesRequest) ProtoMessage() {}
func (x *RebuildIndexesRequest) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[4]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use RebuildIndexesRequest.ProtoReflect.Descriptor instead.
func (*RebuildIndexesRequest) Descriptor() ([]byte, []int) {
return file_search_proto_rawDescGZIP(), []int{4}
}
func (x *RebuildIndexesRequest) GetNamespace() string {
if x != nil {
return x.Namespace
}
return ""
}
func (x *RebuildIndexesRequest) GetKeys() []*ResourceKey {
if x != nil {
return x.Keys
}
return nil
}
type RebuildIndexesResponse struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Total count of rebuilt indexes
RebuildCount int64 `protobuf:"varint,1,opt,name=rebuildCount,proto3" json:"rebuildCount,omitempty"`
// Result message
Details string `protobuf:"bytes,2,opt,name=details,proto3" json:"details,omitempty"`
// Error details
Error *ErrorResult `protobuf:"bytes,3,opt,name=error,proto3" json:"error,omitempty"`
unknownFields protoimpl.UnknownFields
sizeCache protoimpl.SizeCache
}
func (x *RebuildIndexesResponse) Reset() {
*x = RebuildIndexesResponse{}
mi := &file_search_proto_msgTypes[5]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
func (x *RebuildIndexesResponse) String() string {
return protoimpl.X.MessageStringOf(x)
}
func (*RebuildIndexesResponse) ProtoMessage() {}
func (x *RebuildIndexesResponse) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[5]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
ms.StoreMessageInfo(mi)
}
return ms
}
return mi.MessageOf(x)
}
// Deprecated: Use RebuildIndexesResponse.ProtoReflect.Descriptor instead.
func (*RebuildIndexesResponse) Descriptor() ([]byte, []int) {
return file_search_proto_rawDescGZIP(), []int{5}
}
func (x *RebuildIndexesResponse) GetRebuildCount() int64 {
if x != nil {
return x.RebuildCount
}
return 0
}
func (x *RebuildIndexesResponse) GetDetails() string {
if x != nil {
return x.Details
}
return ""
}
func (x *RebuildIndexesResponse) GetError() *ErrorResult {
if x != nil {
return x.Error
}
return nil
}
type ResourceStatsResponse_Stats struct {
state protoimpl.MessageState `protogen:"open.v1"`
// Resource group
@@ -400,7 +517,7 @@ type ResourceStatsResponse_Stats struct {
func (x *ResourceStatsResponse_Stats) Reset() {
*x = ResourceStatsResponse_Stats{}
mi := &file_search_proto_msgTypes[4]
mi := &file_search_proto_msgTypes[6]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -412,7 +529,7 @@ func (x *ResourceStatsResponse_Stats) String() string {
func (*ResourceStatsResponse_Stats) ProtoMessage() {}
func (x *ResourceStatsResponse_Stats) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[4]
mi := &file_search_proto_msgTypes[6]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -459,7 +576,7 @@ type ResourceSearchRequest_Sort struct {
func (x *ResourceSearchRequest_Sort) Reset() {
*x = ResourceSearchRequest_Sort{}
mi := &file_search_proto_msgTypes[5]
mi := &file_search_proto_msgTypes[7]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -471,7 +588,7 @@ func (x *ResourceSearchRequest_Sort) String() string {
func (*ResourceSearchRequest_Sort) ProtoMessage() {}
func (x *ResourceSearchRequest_Sort) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[5]
mi := &file_search_proto_msgTypes[7]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -511,7 +628,7 @@ type ResourceSearchRequest_Facet struct {
func (x *ResourceSearchRequest_Facet) Reset() {
*x = ResourceSearchRequest_Facet{}
mi := &file_search_proto_msgTypes[6]
mi := &file_search_proto_msgTypes[8]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -523,7 +640,7 @@ func (x *ResourceSearchRequest_Facet) String() string {
func (*ResourceSearchRequest_Facet) ProtoMessage() {}
func (x *ResourceSearchRequest_Facet) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[6]
mi := &file_search_proto_msgTypes[8]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -568,7 +685,7 @@ type ResourceSearchResponse_Facet struct {
func (x *ResourceSearchResponse_Facet) Reset() {
*x = ResourceSearchResponse_Facet{}
mi := &file_search_proto_msgTypes[8]
mi := &file_search_proto_msgTypes[10]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -580,7 +697,7 @@ func (x *ResourceSearchResponse_Facet) String() string {
func (*ResourceSearchResponse_Facet) ProtoMessage() {}
func (x *ResourceSearchResponse_Facet) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[8]
mi := &file_search_proto_msgTypes[10]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -634,7 +751,7 @@ type ResourceSearchResponse_TermFacet struct {
func (x *ResourceSearchResponse_TermFacet) Reset() {
*x = ResourceSearchResponse_TermFacet{}
mi := &file_search_proto_msgTypes[9]
mi := &file_search_proto_msgTypes[11]
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
ms.StoreMessageInfo(mi)
}
@@ -646,7 +763,7 @@ func (x *ResourceSearchResponse_TermFacet) String() string {
func (*ResourceSearchResponse_TermFacet) ProtoMessage() {}
func (x *ResourceSearchResponse_TermFacet) ProtoReflect() protoreflect.Message {
mi := &file_search_proto_msgTypes[9]
mi := &file_search_proto_msgTypes[11]
if x != nil {
ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x))
if ms.LoadMessageInfo() == nil {
@@ -780,22 +897,42 @@ var file_search_proto_rawDesc = string([]byte{
0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x26, 0x2e, 0x72,
0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x46,
0x61, 0x63, 0x65, 0x74, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x32,
0xa9, 0x01, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x49, 0x6e, 0x64, 0x65,
0x78, 0x12, 0x4b, 0x0a, 0x06, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x12, 0x1f, 0x2e, 0x72, 0x65,
0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53,
0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x72,
0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4b,
0x0a, 0x08, 0x47, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x73, 0x12, 0x1e, 0x2e, 0x72, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x74,
0x61, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x72, 0x65, 0x73,
0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x74,
0x61, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x3b, 0x5a, 0x39, 0x67,
0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x72, 0x61, 0x66, 0x61, 0x6e,
0x61, 0x2f, 0x67, 0x72, 0x61, 0x66, 0x61, 0x6e, 0x61, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x73, 0x74,
0x6f, 0x72, 0x61, 0x67, 0x65, 0x2f, 0x75, 0x6e, 0x69, 0x66, 0x69, 0x65, 0x64, 0x2f, 0x72, 0x65,
0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
0x61, 0x63, 0x65, 0x74, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22,
0x60, 0x0a, 0x15, 0x52, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x65,
0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x1c, 0x0a, 0x09, 0x6e, 0x61, 0x6d, 0x65,
0x73, 0x70, 0x61, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x6e, 0x61, 0x6d,
0x65, 0x73, 0x70, 0x61, 0x63, 0x65, 0x12, 0x29, 0x0a, 0x04, 0x6b, 0x65, 0x79, 0x73, 0x18, 0x02,
0x20, 0x03, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e,
0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x4b, 0x65, 0x79, 0x52, 0x04, 0x6b, 0x65, 0x79,
0x73, 0x22, 0x83, 0x01, 0x0a, 0x16, 0x52, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x49, 0x6e, 0x64,
0x65, 0x78, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x22, 0x0a, 0x0c,
0x72, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x01, 0x20, 0x01,
0x28, 0x03, 0x52, 0x0c, 0x72, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x43, 0x6f, 0x75, 0x6e, 0x74,
0x12, 0x18, 0x0a, 0x07, 0x64, 0x65, 0x74, 0x61, 0x69, 0x6c, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28,
0x09, 0x52, 0x07, 0x64, 0x65, 0x74, 0x61, 0x69, 0x6c, 0x73, 0x12, 0x2b, 0x0a, 0x05, 0x65, 0x72,
0x72, 0x6f, 0x72, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x72, 0x65, 0x73, 0x6f,
0x75, 0x72, 0x63, 0x65, 0x2e, 0x45, 0x72, 0x72, 0x6f, 0x72, 0x52, 0x65, 0x73, 0x75, 0x6c, 0x74,
0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x32, 0xfe, 0x01, 0x0a, 0x0d, 0x52, 0x65, 0x73, 0x6f,
0x75, 0x72, 0x63, 0x65, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x12, 0x4b, 0x0a, 0x06, 0x53, 0x65, 0x61,
0x72, 0x63, 0x68, 0x12, 0x1f, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52,
0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65, 0x71,
0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e,
0x52, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x65, 0x61, 0x72, 0x63, 0x68, 0x52, 0x65,
0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4b, 0x0a, 0x08, 0x47, 0x65, 0x74, 0x53, 0x74, 0x61,
0x74, 0x73, 0x12, 0x1e, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65,
0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x74, 0x61, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65,
0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x52, 0x65,
0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x53, 0x74, 0x61, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f,
0x6e, 0x73, 0x65, 0x12, 0x53, 0x0a, 0x0e, 0x52, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x49, 0x6e,
0x64, 0x65, 0x78, 0x65, 0x73, 0x12, 0x1f, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65,
0x2e, 0x52, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x65, 0x73, 0x52,
0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x72, 0x65, 0x73, 0x6f, 0x75, 0x72, 0x63,
0x65, 0x2e, 0x52, 0x65, 0x62, 0x75, 0x69, 0x6c, 0x64, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x65, 0x73,
0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x3b, 0x5a, 0x39, 0x67, 0x69, 0x74, 0x68,
0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x72, 0x61, 0x66, 0x61, 0x6e, 0x61, 0x2f, 0x67,
0x72, 0x61, 0x66, 0x61, 0x6e, 0x61, 0x2f, 0x70, 0x6b, 0x67, 0x2f, 0x73, 0x74, 0x6f, 0x72, 0x61,
0x67, 0x65, 0x2f, 0x75, 0x6e, 0x69, 0x66, 0x69, 0x65, 0x64, 0x2f, 0x72, 0x65, 0x73, 0x6f, 0x75,
0x72, 0x63, 0x65, 0x70, 0x62, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
})
var (
@@ -810,47 +947,53 @@ func file_search_proto_rawDescGZIP() []byte {
return file_search_proto_rawDescData
}
var file_search_proto_msgTypes = make([]protoimpl.MessageInfo, 11)
var file_search_proto_msgTypes = make([]protoimpl.MessageInfo, 13)
var file_search_proto_goTypes = []any{
(*ResourceStatsRequest)(nil), // 0: resource.ResourceStatsRequest
(*ResourceStatsResponse)(nil), // 1: resource.ResourceStatsResponse
(*ResourceSearchRequest)(nil), // 2: resource.ResourceSearchRequest
(*ResourceSearchResponse)(nil), // 3: resource.ResourceSearchResponse
(*ResourceStatsResponse_Stats)(nil), // 4: resource.ResourceStatsResponse.Stats
(*ResourceSearchRequest_Sort)(nil), // 5: resource.ResourceSearchRequest.Sort
(*ResourceSearchRequest_Facet)(nil), // 6: resource.ResourceSearchRequest.Facet
nil, // 7: resource.ResourceSearchRequest.FacetEntry
(*ResourceSearchResponse_Facet)(nil), // 8: resource.ResourceSearchResponse.Facet
(*ResourceSearchResponse_TermFacet)(nil), // 9: resource.ResourceSearchResponse.TermFacet
nil, // 10: resource.ResourceSearchResponse.FacetEntry
(*ErrorResult)(nil), // 11: resource.ErrorResult
(*ListOptions)(nil), // 12: resource.ListOptions
(*ResourceKey)(nil), // 13: resource.ResourceKey
(*ResourceTable)(nil), // 14: resource.ResourceTable
(*RebuildIndexesRequest)(nil), // 4: resource.RebuildIndexesRequest
(*RebuildIndexesResponse)(nil), // 5: resource.RebuildIndexesResponse
(*ResourceStatsResponse_Stats)(nil), // 6: resource.ResourceStatsResponse.Stats
(*ResourceSearchRequest_Sort)(nil), // 7: resource.ResourceSearchRequest.Sort
(*ResourceSearchRequest_Facet)(nil), // 8: resource.ResourceSearchRequest.Facet
nil, // 9: resource.ResourceSearchRequest.FacetEntry
(*ResourceSearchResponse_Facet)(nil), // 10: resource.ResourceSearchResponse.Facet
(*ResourceSearchResponse_TermFacet)(nil), // 11: resource.ResourceSearchResponse.TermFacet
nil, // 12: resource.ResourceSearchResponse.FacetEntry
(*ErrorResult)(nil), // 13: resource.ErrorResult
(*ListOptions)(nil), // 14: resource.ListOptions
(*ResourceKey)(nil), // 15: resource.ResourceKey
(*ResourceTable)(nil), // 16: resource.ResourceTable
}
var file_search_proto_depIdxs = []int32{
11, // 0: resource.ResourceStatsResponse.error:type_name -> resource.ErrorResult
4, // 1: resource.ResourceStatsResponse.stats:type_name -> resource.ResourceStatsResponse.Stats
12, // 2: resource.ResourceSearchRequest.options:type_name -> resource.ListOptions
13, // 3: resource.ResourceSearchRequest.federated:type_name -> resource.ResourceKey
5, // 4: resource.ResourceSearchRequest.sortBy:type_name -> resource.ResourceSearchRequest.Sort
7, // 5: resource.ResourceSearchRequest.facet:type_name -> resource.ResourceSearchRequest.FacetEntry
11, // 6: resource.ResourceSearchResponse.error:type_name -> resource.ErrorResult
13, // 7: resource.ResourceSearchResponse.key:type_name -> resource.ResourceKey
14, // 8: resource.ResourceSearchResponse.results:type_name -> resource.ResourceTable
10, // 9: resource.ResourceSearchResponse.facet:type_name -> resource.ResourceSearchResponse.FacetEntry
6, // 10: resource.ResourceSearchRequest.FacetEntry.value:type_name -> resource.ResourceSearchRequest.Facet
9, // 11: resource.ResourceSearchResponse.Facet.terms:type_name -> resource.ResourceSearchResponse.TermFacet
8, // 12: resource.ResourceSearchResponse.FacetEntry.value:type_name -> resource.ResourceSearchResponse.Facet
2, // 13: resource.ResourceIndex.Search:input_type -> resource.ResourceSearchRequest
0, // 14: resource.ResourceIndex.GetStats:input_type -> resource.ResourceStatsRequest
3, // 15: resource.ResourceIndex.Search:output_type -> resource.ResourceSearchResponse
1, // 16: resource.ResourceIndex.GetStats:output_type -> resource.ResourceStatsResponse
15, // [15:17] is the sub-list for method output_type
13, // [13:15] is the sub-list for method input_type
13, // [13:13] is the sub-list for extension type_name
13, // [13:13] is the sub-list for extension extendee
0, // [0:13] is the sub-list for field type_name
13, // 0: resource.ResourceStatsResponse.error:type_name -> resource.ErrorResult
6, // 1: resource.ResourceStatsResponse.stats:type_name -> resource.ResourceStatsResponse.Stats
14, // 2: resource.ResourceSearchRequest.options:type_name -> resource.ListOptions
15, // 3: resource.ResourceSearchRequest.federated:type_name -> resource.ResourceKey
7, // 4: resource.ResourceSearchRequest.sortBy:type_name -> resource.ResourceSearchRequest.Sort
9, // 5: resource.ResourceSearchRequest.facet:type_name -> resource.ResourceSearchRequest.FacetEntry
13, // 6: resource.ResourceSearchResponse.error:type_name -> resource.ErrorResult
15, // 7: resource.ResourceSearchResponse.key:type_name -> resource.ResourceKey
16, // 8: resource.ResourceSearchResponse.results:type_name -> resource.ResourceTable
12, // 9: resource.ResourceSearchResponse.facet:type_name -> resource.ResourceSearchResponse.FacetEntry
15, // 10: resource.RebuildIndexesRequest.keys:type_name -> resource.ResourceKey
13, // 11: resource.RebuildIndexesResponse.error:type_name -> resource.ErrorResult
8, // 12: resource.ResourceSearchRequest.FacetEntry.value:type_name -> resource.ResourceSearchRequest.Facet
11, // 13: resource.ResourceSearchResponse.Facet.terms:type_name -> resource.ResourceSearchResponse.TermFacet
10, // 14: resource.ResourceSearchResponse.FacetEntry.value:type_name -> resource.ResourceSearchResponse.Facet
2, // 15: resource.ResourceIndex.Search:input_type -> resource.ResourceSearchRequest
0, // 16: resource.ResourceIndex.GetStats:input_type -> resource.ResourceStatsRequest
4, // 17: resource.ResourceIndex.RebuildIndexes:input_type -> resource.RebuildIndexesRequest
3, // 18: resource.ResourceIndex.Search:output_type -> resource.ResourceSearchResponse
1, // 19: resource.ResourceIndex.GetStats:output_type -> resource.ResourceStatsResponse
5, // 20: resource.ResourceIndex.RebuildIndexes:output_type -> resource.RebuildIndexesResponse
18, // [18:21] is the sub-list for method output_type
15, // [15:18] is the sub-list for method input_type
15, // [15:15] is the sub-list for extension type_name
15, // [15:15] is the sub-list for extension extendee
0, // [0:15] is the sub-list for field type_name
}
func init() { file_search_proto_init() }
@@ -865,7 +1008,7 @@ func file_search_proto_init() {
GoPackagePath: reflect.TypeOf(x{}).PkgPath(),
RawDescriptor: unsafe.Slice(unsafe.StringData(file_search_proto_rawDesc), len(file_search_proto_rawDesc)),
NumEnums: 0,
NumMessages: 11,
NumMessages: 13,
NumExtensions: 0,
NumServices: 1,
},
@@ -19,8 +19,9 @@ import (
const _ = grpc.SupportPackageIsVersion8
const (
ResourceIndex_Search_FullMethodName = "/resource.ResourceIndex/Search"
ResourceIndex_GetStats_FullMethodName = "/resource.ResourceIndex/GetStats"
ResourceIndex_Search_FullMethodName = "/resource.ResourceIndex/Search"
ResourceIndex_GetStats_FullMethodName = "/resource.ResourceIndex/GetStats"
ResourceIndex_RebuildIndexes_FullMethodName = "/resource.ResourceIndex/RebuildIndexes"
)
// ResourceIndexClient is the client API for ResourceIndex service.
@@ -33,6 +34,7 @@ type ResourceIndexClient interface {
Search(ctx context.Context, in *ResourceSearchRequest, opts ...grpc.CallOption) (*ResourceSearchResponse, error)
// Get the resource stats
GetStats(ctx context.Context, in *ResourceStatsRequest, opts ...grpc.CallOption) (*ResourceStatsResponse, error)
RebuildIndexes(ctx context.Context, in *RebuildIndexesRequest, opts ...grpc.CallOption) (*RebuildIndexesResponse, error)
}
type resourceIndexClient struct {
@@ -63,6 +65,16 @@ func (c *resourceIndexClient) GetStats(ctx context.Context, in *ResourceStatsReq
return out, nil
}
func (c *resourceIndexClient) RebuildIndexes(ctx context.Context, in *RebuildIndexesRequest, opts ...grpc.CallOption) (*RebuildIndexesResponse, error) {
cOpts := append([]grpc.CallOption{grpc.StaticMethod()}, opts...)
out := new(RebuildIndexesResponse)
err := c.cc.Invoke(ctx, ResourceIndex_RebuildIndexes_FullMethodName, in, out, cOpts...)
if err != nil {
return nil, err
}
return out, nil
}
// ResourceIndexServer is the server API for ResourceIndex service.
// All implementations should embed UnimplementedResourceIndexServer
// for forward compatibility
@@ -73,6 +85,7 @@ type ResourceIndexServer interface {
Search(context.Context, *ResourceSearchRequest) (*ResourceSearchResponse, error)
// Get the resource stats
GetStats(context.Context, *ResourceStatsRequest) (*ResourceStatsResponse, error)
RebuildIndexes(context.Context, *RebuildIndexesRequest) (*RebuildIndexesResponse, error)
}
// UnimplementedResourceIndexServer should be embedded to have forward compatible implementations.
@@ -85,6 +98,9 @@ func (UnimplementedResourceIndexServer) Search(context.Context, *ResourceSearchR
func (UnimplementedResourceIndexServer) GetStats(context.Context, *ResourceStatsRequest) (*ResourceStatsResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method GetStats not implemented")
}
func (UnimplementedResourceIndexServer) RebuildIndexes(context.Context, *RebuildIndexesRequest) (*RebuildIndexesResponse, error) {
return nil, status.Errorf(codes.Unimplemented, "method RebuildIndexes not implemented")
}
// UnsafeResourceIndexServer may be embedded to opt out of forward compatibility for this service.
// Use of this interface is not recommended, as added methods to ResourceIndexServer will
@@ -133,6 +149,24 @@ func _ResourceIndex_GetStats_Handler(srv interface{}, ctx context.Context, dec f
return interceptor(ctx, in, info, handler)
}
func _ResourceIndex_RebuildIndexes_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) {
in := new(RebuildIndexesRequest)
if err := dec(in); err != nil {
return nil, err
}
if interceptor == nil {
return srv.(ResourceIndexServer).RebuildIndexes(ctx, in)
}
info := &grpc.UnaryServerInfo{
Server: srv,
FullMethod: ResourceIndex_RebuildIndexes_FullMethodName,
}
handler := func(ctx context.Context, req interface{}) (interface{}, error) {
return srv.(ResourceIndexServer).RebuildIndexes(ctx, req.(*RebuildIndexesRequest))
}
return interceptor(ctx, in, info, handler)
}
// ResourceIndex_ServiceDesc is the grpc.ServiceDesc for ResourceIndex service.
// It's only intended for direct use with grpc.RegisterService,
// and not to be introspected or modified (even as a copy)
@@ -148,6 +182,10 @@ var ResourceIndex_ServiceDesc = grpc.ServiceDesc{
MethodName: "GetStats",
Handler: _ResourceIndex_GetStats_Handler,
},
{
MethodName: "RebuildIndexes",
Handler: _ResourceIndex_RebuildIndexes_Handler,
},
},
Streams: []grpc.StreamDesc{},
Metadata: "search.proto",
+7 -1
View File
@@ -994,7 +994,9 @@ func (b *backend) GetResourceLastImportTimes(ctx context.Context) iter.Seq2[reso
b.lastImportTimeDeletionTime.Store(now)
}
rows, err := dbutil.QueryRows(ctx, b.db, sqlResourceLastImportTimeQuery, &sqlResourceLastImportTimeQueryRequest{SQLTemplate: sqltemplate.New(b.dialect)})
rows, err := dbutil.QueryRows(ctx, b.db, sqlResourceLastImportTimeQuery, &sqlResourceLastImportTimeQueryRequest{
SQLTemplate: sqltemplate.New(b.dialect),
})
if err != nil {
return func(yield func(resource.ResourceLastImportTime, error) bool) {
yield(resource.ResourceLastImportTime{}, err)
@@ -1037,3 +1039,7 @@ func (b *backend) GetResourceLastImportTimes(ctx context.Context) iter.Seq2[reso
}
}
}
func (b *backend) RebuildIndexes(ctx context.Context, req *resourcepb.RebuildIndexesRequest) (*resourcepb.RebuildIndexesResponse, error) {
return nil, fmt.Errorf("rebuild indexes not supported by unistore sql backend")
}
@@ -1,85 +0,0 @@
package postgres
import (
"fmt"
"sync"
)
// locker is a named reader/writer mutual exclusion lock.
// The lock for each particular key can be held by an arbitrary number of readers or a single writer.
type locker struct {
locks map[any]*sync.RWMutex
locksRW *sync.RWMutex
}
func newLocker() *locker {
return &locker{
locks: make(map[any]*sync.RWMutex),
locksRW: new(sync.RWMutex),
}
}
// Lock locks named rw mutex with specified key for writing.
// If the lock with the same key is already locked for reading or writing,
// Lock blocks until the lock is available.
func (lkr *locker) Lock(key any) {
lk, ok := lkr.getLock(key)
if !ok {
lk = lkr.newLock(key)
}
lk.Lock()
}
// Unlock unlocks named rw mutex with specified key for writing. It is a run-time error if rw is
// not locked for writing on entry to Unlock.
func (lkr *locker) Unlock(key any) {
lk, ok := lkr.getLock(key)
if !ok {
panic(fmt.Errorf("lock for key '%s' not initialized", key))
}
lk.Unlock()
}
// RLock locks named rw mutex with specified key for reading.
//
// It should not be used for recursive read locking for the same key; a blocked Lock
// call excludes new readers from acquiring the lock. See the
// documentation on the golang RWMutex type.
func (lkr *locker) RLock(key any) {
lk, ok := lkr.getLock(key)
if !ok {
lk = lkr.newLock(key)
}
lk.RLock()
}
// RUnlock undoes a single RLock call for specified key;
// it does not affect other simultaneous readers of locker for specified key.
// It is a run-time error if locker for specified key is not locked for reading
func (lkr *locker) RUnlock(key any) {
lk, ok := lkr.getLock(key)
if !ok {
panic(fmt.Errorf("lock for key '%s' not initialized", key))
}
lk.RUnlock()
}
func (lkr *locker) newLock(key any) *sync.RWMutex {
lkr.locksRW.Lock()
defer lkr.locksRW.Unlock()
if lk, ok := lkr.locks[key]; ok {
return lk
}
lk := new(sync.RWMutex)
lkr.locks[key] = lk
return lk
}
func (lkr *locker) getLock(key any) (*sync.RWMutex, bool) {
lkr.locksRW.RLock()
defer lkr.locksRW.RUnlock()
lock, ok := lkr.locks[key]
return lock, ok
}
@@ -1,63 +0,0 @@
package postgres
import (
"sync"
"testing"
"time"
"github.com/grafana/grafana/pkg/util/testutil"
"github.com/stretchr/testify/require"
)
func TestIntegrationLocker(t *testing.T) {
testutil.SkipIntegrationTestInShortMode(t)
const notUpdated = "not_updated"
const atThread1 = "at_thread_1"
const atThread2 = "at_thread_2"
t.Run("Should lock for same keys", func(t *testing.T) {
updated := notUpdated
locker := newLocker()
locker.Lock(1)
var wg sync.WaitGroup
wg.Add(1)
defer func() {
locker.Unlock(1)
wg.Wait()
}()
go func() {
locker.RLock(1)
defer func() {
locker.RUnlock(1)
wg.Done()
}()
require.Equal(t, atThread1, updated, "Value should be updated in different thread")
updated = atThread2
}()
time.Sleep(time.Millisecond * 10)
require.Equal(t, notUpdated, updated, "Value should not be updated in different thread")
updated = atThread1
})
t.Run("Should not lock for different keys", func(t *testing.T) {
updated := notUpdated
locker := newLocker()
locker.Lock(1)
defer locker.Unlock(1)
var wg sync.WaitGroup
wg.Add(1)
go func() {
locker.RLock(2)
defer func() {
locker.RUnlock(2)
wg.Done()
}()
require.Equal(t, notUpdated, updated, "Value should not be updated in different thread")
updated = atThread2
}()
wg.Wait()
require.Equal(t, atThread2, updated, "Value should be updated in different thread")
updated = atThread1
})
}
@@ -1,117 +0,0 @@
package pgx
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"strings"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
)
func (e *DataSourceHandler) CheckHealth(ctx context.Context, req *backend.CheckHealthRequest) (*backend.CheckHealthResult, error) {
err := e.Ping(ctx)
if err != nil {
logCheckHealthError(ctx, e.dsInfo, err)
if strings.EqualFold(req.PluginContext.User.Role, "Admin") {
return ErrToHealthCheckResult(err)
}
errResponse := &backend.CheckHealthResult{
Status: backend.HealthStatusError,
Message: e.TransformQueryError(e.log, err).Error(),
}
return errResponse, nil
}
return &backend.CheckHealthResult{Status: backend.HealthStatusOk, Message: "Database Connection OK"}, nil
}
// ErrToHealthCheckResult converts error into user friendly health check message
// This should be called with non nil error. If the err parameter is empty, we will send Internal Server Error
func ErrToHealthCheckResult(err error) (*backend.CheckHealthResult, error) {
if err == nil {
return &backend.CheckHealthResult{Status: backend.HealthStatusError, Message: "Internal Server Error"}, nil
}
res := &backend.CheckHealthResult{Status: backend.HealthStatusError, Message: err.Error()}
details := map[string]string{
"verboseMessage": err.Error(),
"errorDetailsLink": "https://grafana.com/docs/grafana/latest/datasources/postgres",
}
var opErr *net.OpError
if errors.As(err, &opErr) {
res.Message = "Network error: Failed to connect to the server"
if opErr != nil && opErr.Err != nil {
errMessage := opErr.Err.Error()
if strings.HasSuffix(opErr.Err.Error(), "no such host") {
errMessage = "no such host"
}
if strings.HasSuffix(opErr.Err.Error(), "unknown port") {
errMessage = "unknown port"
}
if strings.HasSuffix(opErr.Err.Error(), "invalid port") {
errMessage = "invalid port"
}
if strings.HasSuffix(opErr.Err.Error(), "missing port in address") {
errMessage = "missing port in address"
}
if strings.HasSuffix(opErr.Err.Error(), "invalid syntax") {
errMessage = "invalid syntax found in the address"
}
res.Message += fmt.Sprintf(". Error message: %s", errMessage)
}
}
if errors.Is(err, sqleng.ErrParsingPostgresURL) {
res.Message = fmt.Sprintf("Connection string error: %s", sqleng.ErrParsingPostgresURL.Error())
if unwrappedErr := errors.Unwrap(err); unwrappedErr != nil {
details["verboseMessage"] = unwrappedErr.Error()
}
}
detailBytes, marshalErr := json.Marshal(details)
if marshalErr != nil {
return res, nil
}
res.JSONDetails = detailBytes
return res, nil
}
func logCheckHealthError(ctx context.Context, dsInfo sqleng.DataSourceInfo, err error) {
logger := log.DefaultLogger.FromContext(ctx)
configSummary := map[string]any{
"config_url_length": len(dsInfo.URL),
"config_user_length": len(dsInfo.User),
"config_database_length": len(dsInfo.Database),
"config_json_data_database_length": len(dsInfo.JsonData.Database),
"config_max_open_conns": dsInfo.JsonData.MaxOpenConns,
"config_max_idle_conns": dsInfo.JsonData.MaxIdleConns,
"config_conn_max_life_time": dsInfo.JsonData.ConnMaxLifetime,
"config_conn_timeout": dsInfo.JsonData.ConnectionTimeout,
"config_timescaledb": dsInfo.JsonData.Timescaledb,
"config_ssl_mode": dsInfo.JsonData.Mode,
"config_tls_configuration_method": dsInfo.JsonData.ConfigurationMethod,
"config_tls_skip_verify": dsInfo.JsonData.TlsSkipVerify,
"config_timezone": dsInfo.JsonData.Timezone,
"config_time_interval": dsInfo.JsonData.TimeInterval,
"config_enable_secure_proxy": dsInfo.JsonData.SecureDSProxy,
"config_allow_clear_text_passwords": dsInfo.JsonData.AllowCleartextPasswords,
"config_authentication_type": dsInfo.JsonData.AuthenticationType,
"config_ssl_root_cert_file_length": len(dsInfo.JsonData.RootCertFile),
"config_ssl_cert_file_length": len(dsInfo.JsonData.CertFile),
"config_ssl_key_file_length": len(dsInfo.JsonData.CertKeyFile),
"config_encrypt_length": len(dsInfo.JsonData.Encrypt),
"config_server_name_length": len(dsInfo.JsonData.Servername),
"config_password_length": len(dsInfo.DecryptedSecureJSONData["password"]),
"config_tls_ca_cert_length": len(dsInfo.DecryptedSecureJSONData["tlsCACert"]),
"config_tls_client_cert_length": len(dsInfo.DecryptedSecureJSONData["tlsClientCert"]),
"config_tls_client_key_length": len(dsInfo.DecryptedSecureJSONData["tlsClientKey"]),
}
configSummaryJSON, marshalError := json.Marshal(configSummary)
if marshalError != nil {
logger.Error("Check health failed", "error", err, "message_type", "ds_config_health_check_error")
return
}
logger.Error("Check health failed", "error", err, "message_type", "ds_config_health_check_error_detailed", "details", string(configSummaryJSON))
}
@@ -1,61 +0,0 @@
package pgx
import (
"errors"
"fmt"
"net"
"testing"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestErrToHealthCheckResult(t *testing.T) {
tests := []struct {
name string
err error
want *backend.CheckHealthResult
}{
{
name: "without error",
want: &backend.CheckHealthResult{Status: backend.HealthStatusError, Message: "Internal Server Error"},
},
{
name: "network error",
err: errors.Join(errors.New("foo"), &net.OpError{Op: "read", Net: "tcp", Err: errors.New("some op")}),
want: &backend.CheckHealthResult{
Status: backend.HealthStatusError,
Message: "Network error: Failed to connect to the server. Error message: some op",
JSONDetails: []byte(`{"errorDetailsLink":"https://grafana.com/docs/grafana/latest/datasources/postgres","verboseMessage":"foo\nread tcp: some op"}`),
},
},
{
name: "regular error",
err: errors.New("internal server error"),
want: &backend.CheckHealthResult{
Status: backend.HealthStatusError,
Message: "internal server error",
JSONDetails: []byte(`{"errorDetailsLink":"https://grafana.com/docs/grafana/latest/datasources/postgres","verboseMessage":"internal server error"}`),
},
},
{
name: "invalid port specifier error",
err: fmt.Errorf("%w %q: %w", sqleng.ErrParsingPostgresURL, `"foo.bar.co"`, errors.New(`strconv.Atoi: parsing "foo.bar.co": invalid syntax`)),
want: &backend.CheckHealthResult{
Status: backend.HealthStatusError,
Message: "Connection string error: error parsing postgres url",
JSONDetails: []byte(`{"errorDetailsLink":"https://grafana.com/docs/grafana/latest/datasources/postgres","verboseMessage":"error parsing postgres url \"\\\"foo.bar.co\\\"\": strconv.Atoi: parsing \"foo.bar.co\": invalid syntax"}`),
},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := ErrToHealthCheckResult(tt.err)
require.Nil(t, err)
assert.Equal(t, string(tt.want.JSONDetails), string(got.JSONDetails))
require.Equal(t, tt.want, got)
})
}
}
@@ -1,836 +0,0 @@
package pgx
import (
"context"
"encoding/json"
"errors"
"fmt"
"net"
"runtime/debug"
"strconv"
"strings"
"sync"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/gtime"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgtype"
"github.com/jackc/pgx/v5/pgxpool"
)
// MetaKeyExecutedQueryString is the key where the executed query should get stored
const MetaKeyExecutedQueryString = "executedQueryString"
// SQLMacroEngine interpolates macros into sql. It takes in the Query to have access to query context and
// timeRange to be able to generate queries that use from and to.
type SQLMacroEngine interface {
Interpolate(query *backend.DataQuery, timeRange backend.TimeRange, sql string) (string, error)
}
// SqlQueryResultTransformer transforms a query result row to RowValues with proper types.
type SqlQueryResultTransformer interface {
// TransformQueryError transforms a query error.
TransformQueryError(logger log.Logger, err error) error
GetConverterList() []sqlutil.StringConverter
}
type JsonData struct {
MaxOpenConns int `json:"maxOpenConns"`
MaxIdleConns int `json:"maxIdleConns"`
ConnMaxLifetime int `json:"connMaxLifetime"`
ConnectionTimeout int `json:"connectionTimeout"`
Timescaledb bool `json:"timescaledb"`
Mode string `json:"sslmode"`
ConfigurationMethod string `json:"tlsConfigurationMethod"`
TlsSkipVerify bool `json:"tlsSkipVerify"`
RootCertFile string `json:"sslRootCertFile"`
CertFile string `json:"sslCertFile"`
CertKeyFile string `json:"sslKeyFile"`
Timezone string `json:"timezone"`
Encrypt string `json:"encrypt"`
Servername string `json:"servername"`
TimeInterval string `json:"timeInterval"`
Database string `json:"database"`
SecureDSProxy bool `json:"enableSecureSocksProxy"`
SecureDSProxyUsername string `json:"secureSocksProxyUsername"`
AllowCleartextPasswords bool `json:"allowCleartextPasswords"`
AuthenticationType string `json:"authenticationType"`
}
type DataPluginConfiguration struct {
DSInfo sqleng.DataSourceInfo
TimeColumnNames []string
MetricColumnTypes []string
RowLimit int64
}
type DataSourceHandler struct {
macroEngine SQLMacroEngine
queryResultTransformer SqlQueryResultTransformer
timeColumnNames []string
metricColumnTypes []string
log log.Logger
dsInfo sqleng.DataSourceInfo
rowLimit int64
userError string
pool *pgxpool.Pool
}
type QueryJson struct {
RawSql string `json:"rawSql"`
Fill bool `json:"fill"`
FillInterval float64 `json:"fillInterval"`
FillMode string `json:"fillMode"`
FillValue float64 `json:"fillValue"`
Format string `json:"format"`
}
func (e *DataSourceHandler) TransformQueryError(logger log.Logger, err error) error {
// OpError is the error type usually returned by functions in the net
// package. It describes the operation, network type, and address of
// an error. We log this error rather than return it to the client
// for security purposes.
var opErr *net.OpError
if errors.As(err, &opErr) {
return fmt.Errorf("failed to connect to server - %s", e.userError)
}
return e.queryResultTransformer.TransformQueryError(logger, err)
}
func NewQueryDataHandler(userFacingDefaultError string, p *pgxpool.Pool, config DataPluginConfiguration, queryResultTransformer SqlQueryResultTransformer,
macroEngine SQLMacroEngine, log log.Logger) (*DataSourceHandler, error) {
queryDataHandler := DataSourceHandler{
queryResultTransformer: queryResultTransformer,
macroEngine: macroEngine,
timeColumnNames: []string{"time"},
log: log,
dsInfo: config.DSInfo,
rowLimit: config.RowLimit,
userError: userFacingDefaultError,
}
if len(config.TimeColumnNames) > 0 {
queryDataHandler.timeColumnNames = config.TimeColumnNames
}
if len(config.MetricColumnTypes) > 0 {
queryDataHandler.metricColumnTypes = config.MetricColumnTypes
}
queryDataHandler.pool = p
return &queryDataHandler, nil
}
type DBDataResponse struct {
dataResponse backend.DataResponse
refID string
}
func (e *DataSourceHandler) Dispose() {
e.log.Debug("Disposing DB...")
if e.pool != nil {
e.pool.Close()
}
e.log.Debug("DB disposed")
}
func (e *DataSourceHandler) Ping(ctx context.Context) error {
return e.pool.Ping(ctx)
}
func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
result := backend.NewQueryDataResponse()
ch := make(chan DBDataResponse, len(req.Queries))
var wg sync.WaitGroup
// Execute each query in a goroutine and wait for them to finish afterwards
for _, query := range req.Queries {
queryjson := QueryJson{
Fill: false,
Format: "time_series",
}
err := json.Unmarshal(query.JSON, &queryjson)
if err != nil {
return nil, backend.DownstreamErrorf("error unmarshal query json: %s", err.Error())
}
// the fill-params are only stored inside this function, during query-interpolation. we do not support
// sending them in "from the outside"
if queryjson.Fill || queryjson.FillInterval != 0.0 || queryjson.FillMode != "" || queryjson.FillValue != 0.0 {
return nil, backend.DownstreamErrorf("query fill-parameters not supported")
}
if queryjson.RawSql == "" {
continue
}
wg.Add(1)
go e.executeQuery(ctx, query, &wg, ch, queryjson)
}
wg.Wait()
// Read results from channels
close(ch)
result.Responses = make(map[string]backend.DataResponse)
for queryResult := range ch {
result.Responses[queryResult.refID] = queryResult.dataResponse
}
return result, nil
}
func (e *DataSourceHandler) handleQueryError(frameErr string, err error, query string, source backend.ErrorSource, ch chan DBDataResponse, queryResult DBDataResponse) {
var emptyFrame data.Frame
emptyFrame.SetMeta(&data.FrameMeta{ExecutedQueryString: query})
if isDownstreamError(err) {
source = backend.ErrorSourceDownstream
}
queryResult.dataResponse.Error = fmt.Errorf("%s: %w", frameErr, err)
queryResult.dataResponse.ErrorSource = source
queryResult.dataResponse.Frames = data.Frames{&emptyFrame}
ch <- queryResult
}
func (e *DataSourceHandler) handlePanic(logger log.Logger, queryResult *DBDataResponse, ch chan DBDataResponse) {
if r := recover(); r != nil {
logger.Error("ExecuteQuery panic", "error", r, "stack", string(debug.Stack()))
if theErr, ok := r.(error); ok {
queryResult.dataResponse.Error = theErr
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else if theErrString, ok := r.(string); ok {
queryResult.dataResponse.Error = errors.New(theErrString)
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else {
queryResult.dataResponse.Error = fmt.Errorf("unexpected error - %s", e.userError)
queryResult.dataResponse.ErrorSource = backend.ErrorSourceDownstream
}
ch <- *queryResult
}
}
// Interpolate provides global macros/substitutions for all sql datasources.
var Interpolate = func(query backend.DataQuery, timeRange backend.TimeRange, timeInterval string, sql string) string {
interval := query.Interval
sql = strings.ReplaceAll(sql, "$__interval_ms", strconv.FormatInt(interval.Milliseconds(), 10))
sql = strings.ReplaceAll(sql, "$__interval", gtime.FormatInterval(interval))
sql = strings.ReplaceAll(sql, "$__unixEpochFrom()", fmt.Sprintf("%d", timeRange.From.UTC().Unix()))
sql = strings.ReplaceAll(sql, "$__unixEpochTo()", fmt.Sprintf("%d", timeRange.To.UTC().Unix()))
return sql
}
func (e *DataSourceHandler) execQuery(ctx context.Context, query string) ([]*pgconn.Result, error) {
c, err := e.pool.Acquire(ctx)
if err != nil {
return nil, backend.DownstreamErrorf("failed to acquire connection: %w", err)
}
defer c.Release()
mrr := c.Conn().PgConn().Exec(ctx, query)
// Close returns the first error that occurred during the MultiResultReader's use. We will log that later.
defer mrr.Close() //nolint:errcheck
return mrr.ReadAll()
}
func (e *DataSourceHandler) executeQuery(queryContext context.Context, query backend.DataQuery, wg *sync.WaitGroup,
ch chan DBDataResponse, queryJSON QueryJson) {
defer wg.Done()
queryResult := DBDataResponse{
dataResponse: backend.DataResponse{},
refID: query.RefID,
}
logger := e.log.FromContext(queryContext)
defer e.handlePanic(logger, &queryResult, ch)
if queryJSON.RawSql == "" {
panic("Query model property rawSql should not be empty at this point")
}
// global substitutions
interpolatedQuery := Interpolate(query, query.TimeRange, e.dsInfo.JsonData.TimeInterval, queryJSON.RawSql)
// data source specific substitutions
interpolatedQuery, err := e.macroEngine.Interpolate(&query, query.TimeRange, interpolatedQuery)
if err != nil {
e.handleQueryError("interpolation failed", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
results, err := e.execQuery(queryContext, interpolatedQuery)
if err != nil {
e.handleQueryError("db query error", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
qm, err := e.newProcessCfg(queryContext, query, results, interpolatedQuery)
if err != nil {
e.handleQueryError("failed to get configurations", err, interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
frame, err := convertResultsToFrame(results, e.rowLimit)
if err != nil {
e.handleQueryError("convert frame from rows error", err, interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
e.processFrame(frame, qm, queryResult, ch, logger)
}
// dataQueryFormat is the type of query.
type dataQueryFormat string
const (
// dataQueryFormatTable identifies a table query (default).
dataQueryFormatTable dataQueryFormat = "table"
// dataQueryFormatSeries identifies a time series query.
dataQueryFormatSeries dataQueryFormat = "time_series"
)
type dataQueryModel struct {
InterpolatedQuery string // property not set until after Interpolate()
Format dataQueryFormat
TimeRange backend.TimeRange
FillMissing *data.FillMissing // property not set until after Interpolate()
Interval time.Duration
columnNames []string
columnTypes []string
timeIndex int
timeEndIndex int
metricIndex int
metricPrefix bool
queryContext context.Context
}
func convertSQLTimeColumnsToEpochMS(frame *data.Frame, qm *dataQueryModel) error {
if qm.timeIndex != -1 {
if err := convertSQLTimeColumnToEpochMS(frame, qm.timeIndex); err != nil {
return fmt.Errorf("%v: %w", "failed to convert time column", err)
}
}
if qm.timeEndIndex != -1 {
if err := convertSQLTimeColumnToEpochMS(frame, qm.timeEndIndex); err != nil {
return fmt.Errorf("%v: %w", "failed to convert timeend column", err)
}
}
return nil
}
func (e *DataSourceHandler) processFrame(frame *data.Frame, qm *dataQueryModel, queryResult DBDataResponse, ch chan DBDataResponse, logger log.Logger) {
if frame.Meta == nil {
frame.Meta = &data.FrameMeta{}
}
frame.Meta.ExecutedQueryString = qm.InterpolatedQuery
// If no rows were returned, clear any previously set `Fields` with a single empty `data.Field` slice.
// Then assign `queryResult.dataResponse.Frames` the current single frame with that single empty Field.
// This assures 1) our visualization doesn't display unwanted empty fields, and also that 2)
// additionally-needed frame data stays intact and is correctly passed to our visulization.
if frame.Rows() == 0 {
frame.Fields = []*data.Field{}
queryResult.dataResponse.Frames = data.Frames{frame}
ch <- queryResult
return
}
if err := convertSQLTimeColumnsToEpochMS(frame, qm); err != nil {
e.handleQueryError("converting time columns failed", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
if qm.Format == dataQueryFormatSeries {
// time series has to have time column
if qm.timeIndex == -1 {
e.handleQueryError("db has no time column", errors.New("time column is missing; make sure your data includes a time column for time series format or switch to a table format that doesn't require it"), qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
// Make sure to name the time field 'Time' to be backward compatible with Grafana pre-v8.
frame.Fields[qm.timeIndex].Name = data.TimeSeriesTimeFieldName
for i := range qm.columnNames {
if i == qm.timeIndex || i == qm.metricIndex {
continue
}
if t := frame.Fields[i].Type(); t == data.FieldTypeString || t == data.FieldTypeNullableString {
continue
}
var err error
if frame, err = convertSQLValueColumnToFloat(frame, i); err != nil {
e.handleQueryError("convert value to float failed", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
}
tsSchema := frame.TimeSeriesSchema()
if tsSchema.Type == data.TimeSeriesTypeLong {
var err error
originalData := frame
frame, err = data.LongToWide(frame, qm.FillMissing)
if err != nil {
e.handleQueryError("failed to convert long to wide series when converting from dataframe", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
// Before 8x, a special metric column was used to name time series. The LongToWide transforms that into a metric label on the value field.
// But that makes series name have both the value column name AND the metric name. So here we are removing the metric label here and moving it to the
// field name to get the same naming for the series as pre v8
if len(originalData.Fields) == 3 {
for _, field := range frame.Fields {
if len(field.Labels) == 1 { // 7x only supported one label
name, ok := field.Labels["metric"]
if ok {
field.Name = name
field.Labels = nil
}
}
}
}
}
if qm.FillMissing != nil {
// we align the start-time
startUnixTime := qm.TimeRange.From.Unix() / int64(qm.Interval.Seconds()) * int64(qm.Interval.Seconds())
alignedTimeRange := backend.TimeRange{
From: time.Unix(startUnixTime, 0),
To: qm.TimeRange.To,
}
var err error
frame, err = sqlutil.ResampleWideFrame(frame, qm.FillMissing, alignedTimeRange, qm.Interval)
if err != nil {
logger.Error("Failed to resample dataframe", "err", err)
frame.AppendNotices(data.Notice{Text: "Failed to resample dataframe", Severity: data.NoticeSeverityWarning})
return
}
}
}
queryResult.dataResponse.Frames = data.Frames{frame}
ch <- queryResult
}
func (e *DataSourceHandler) newProcessCfg(queryContext context.Context, query backend.DataQuery,
results []*pgconn.Result, interpolatedQuery string) (*dataQueryModel, error) {
columnNames := []string{}
columnTypes := []string{}
// The results will contain column information in the metadata
for _, result := range results {
// Get column names from the result metadata
for _, field := range result.FieldDescriptions {
columnNames = append(columnNames, field.Name)
pqtype, ok := pgtype.NewMap().TypeForOID(field.DataTypeOID)
if !ok {
// Handle special cases for field types
switch field.DataTypeOID {
case pgtype.TimetzOID:
columnTypes = append(columnTypes, "timetz")
case 790:
columnTypes = append(columnTypes, "money")
default:
columnTypes = append(columnTypes, "unknown")
}
} else {
columnTypes = append(columnTypes, pqtype.Name)
}
}
}
qm := &dataQueryModel{
columnTypes: columnTypes,
columnNames: columnNames,
timeIndex: -1,
timeEndIndex: -1,
metricIndex: -1,
metricPrefix: false,
queryContext: queryContext,
}
queryJSON := QueryJson{}
err := json.Unmarshal(query.JSON, &queryJSON)
if err != nil {
return nil, err
}
if queryJSON.Fill {
qm.FillMissing = &data.FillMissing{}
qm.Interval = time.Duration(queryJSON.FillInterval * float64(time.Second))
switch strings.ToLower(queryJSON.FillMode) {
case "null":
qm.FillMissing.Mode = data.FillModeNull
case "previous":
qm.FillMissing.Mode = data.FillModePrevious
case "value":
qm.FillMissing.Mode = data.FillModeValue
qm.FillMissing.Value = queryJSON.FillValue
default:
}
}
qm.TimeRange.From = query.TimeRange.From.UTC()
qm.TimeRange.To = query.TimeRange.To.UTC()
// Default to time_series if no format is provided
switch queryJSON.Format {
case "table":
qm.Format = dataQueryFormatTable
case "time_series":
fallthrough
default:
qm.Format = dataQueryFormatSeries
}
for i, col := range qm.columnNames {
for _, tc := range e.timeColumnNames {
if col == tc {
qm.timeIndex = i
break
}
}
if qm.Format == dataQueryFormatTable && strings.EqualFold(col, "timeend") {
qm.timeEndIndex = i
continue
}
switch col {
case "metric":
qm.metricIndex = i
default:
if qm.metricIndex == -1 {
columnType := qm.columnTypes[i]
for _, mct := range e.metricColumnTypes {
if columnType == mct {
qm.metricIndex = i
continue
}
}
}
}
}
qm.InterpolatedQuery = interpolatedQuery
return qm, nil
}
func convertResultsToFrame(results []*pgconn.Result, rowLimit int64) (*data.Frame, error) {
m := pgtype.NewMap()
// Find the first SELECT result to establish the frame structure
var firstSelectResult *pgconn.Result
for _, result := range results {
if result.CommandTag.Select() {
firstSelectResult = result
break
}
}
// If no SELECT results found, return empty frame
if firstSelectResult == nil {
return data.NewFrame(""), nil
}
// Create frame structure based on the first SELECT result
fields := make(data.Fields, len(firstSelectResult.FieldDescriptions))
fieldTypes, err := getFieldTypesFromDescriptions(firstSelectResult.FieldDescriptions, m)
if err != nil {
return nil, err
}
for i, v := range firstSelectResult.FieldDescriptions {
fields[i] = data.NewFieldFromFieldType(fieldTypes[i], 0)
fields[i].Name = v.Name
}
frame := *data.NewFrame("", fields...)
// Process all SELECT results, but validate column compatibility
for _, result := range results {
// Skip non-select statements
if !result.CommandTag.Select() {
continue
}
// Validate that this result has the same structure as the frame
if len(result.FieldDescriptions) != len(frame.Fields) {
return nil, fmt.Errorf("incompatible result structure: expected %d columns, got %d columns",
len(frame.Fields), len(result.FieldDescriptions))
}
// Validate column names and types match
for i, fd := range result.FieldDescriptions {
if fd.Name != frame.Fields[i].Name {
return nil, fmt.Errorf("column name mismatch at position %d: expected %q, got %q",
i, frame.Fields[i].Name, fd.Name)
}
}
fieldDescriptions := result.FieldDescriptions
for rowIdx := range result.Rows {
if rowIdx == int(rowLimit) {
frame.AppendNotices(data.Notice{
Severity: data.NoticeSeverityWarning,
Text: fmt.Sprintf("Results have been limited to %v because the SQL row limit was reached", rowLimit),
})
break
}
row := make([]any, len(fieldDescriptions))
for colIdx, fd := range fieldDescriptions {
rawValue := result.Rows[rowIdx][colIdx]
if rawValue == nil {
row[colIdx] = nil
continue
}
convertedValue, err := convertPostgresValue(rawValue, fd, m)
if err != nil {
return nil, err
}
row[colIdx] = convertedValue
}
// Validate row length matches frame field count before appending
if len(row) != len(frame.Fields) {
return nil, fmt.Errorf("row data length mismatch: expected %d values, got %d values",
len(frame.Fields), len(row))
}
frame.AppendRow(row...)
}
}
return &frame, nil
}
// convertPostgresValue converts a raw PostgreSQL value to the appropriate Go type
func convertPostgresValue(rawValue []byte, fd pgconn.FieldDescription, m *pgtype.Map) (interface{}, error) {
dataTypeOID := fd.DataTypeOID
format := fd.Format
// Convert based on type
switch fd.DataTypeOID {
case pgtype.Int2OID:
var d *int16
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.Int4OID:
var d *int32
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.Int8OID:
var d *int64
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.NumericOID, pgtype.Float8OID, pgtype.Float4OID:
var d *float64
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.BoolOID:
var d *bool
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.ByteaOID:
d, err := pgtype.ByteaCodec.DecodeValue(pgtype.ByteaCodec{}, m, dataTypeOID, format, rawValue)
if err != nil {
return nil, err
}
str := string(d.([]byte))
return &str, nil
case pgtype.TimestampOID, pgtype.TimestamptzOID, pgtype.DateOID:
var d *time.Time
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.TimeOID, pgtype.TimetzOID:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.JSONOID, pgtype.JSONBOID:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
// Handle null JSON values
if d == nil {
return nil, nil
}
j := json.RawMessage(*d)
return &j, nil
default:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
}
}
func getFieldTypesFromDescriptions(fieldDescriptions []pgconn.FieldDescription, m *pgtype.Map) ([]data.FieldType, error) {
fieldTypes := make([]data.FieldType, len(fieldDescriptions))
for i, v := range fieldDescriptions {
typeName, ok := m.TypeForOID(v.DataTypeOID)
if !ok {
fieldTypes[i] = data.FieldTypeNullableString
} else {
switch typeName.Name {
case "int2":
fieldTypes[i] = data.FieldTypeNullableInt16
case "int4":
fieldTypes[i] = data.FieldTypeNullableInt32
case "int8":
fieldTypes[i] = data.FieldTypeNullableInt64
case "float4", "float8", "numeric":
fieldTypes[i] = data.FieldTypeNullableFloat64
case "bool":
fieldTypes[i] = data.FieldTypeNullableBool
case "timestamptz", "timestamp", "date":
fieldTypes[i] = data.FieldTypeNullableTime
case "json", "jsonb":
fieldTypes[i] = data.FieldTypeNullableJSON
default:
fieldTypes[i] = data.FieldTypeNullableString
}
}
}
return fieldTypes, nil
}
// convertSQLTimeColumnToEpochMS converts column named time to unix timestamp in milliseconds
// to make native datetime types and epoch dates work in annotation and table queries.
func convertSQLTimeColumnToEpochMS(frame *data.Frame, timeIndex int) error {
if timeIndex < 0 || timeIndex >= len(frame.Fields) {
return fmt.Errorf("timeIndex %d is out of range", timeIndex)
}
origin := frame.Fields[timeIndex]
valueType := origin.Type()
if valueType == data.FieldTypeTime || valueType == data.FieldTypeNullableTime {
return nil
}
newField := data.NewFieldFromFieldType(data.FieldTypeNullableTime, 0)
newField.Name = origin.Name
newField.Labels = origin.Labels
valueLength := origin.Len()
for i := 0; i < valueLength; i++ {
v, err := origin.NullableFloatAt(i)
if err != nil {
return fmt.Errorf("unable to convert data to a time field")
}
if v == nil {
newField.Append(nil)
} else {
timestamp := time.Unix(0, int64(epochPrecisionToMS(*v))*int64(time.Millisecond))
newField.Append(&timestamp)
}
}
frame.Fields[timeIndex] = newField
return nil
}
// convertSQLValueColumnToFloat converts timeseries value column to float.
func convertSQLValueColumnToFloat(frame *data.Frame, Index int) (*data.Frame, error) {
if Index < 0 || Index >= len(frame.Fields) {
return frame, fmt.Errorf("metricIndex %d is out of range", Index)
}
origin := frame.Fields[Index]
valueType := origin.Type()
if valueType == data.FieldTypeFloat64 || valueType == data.FieldTypeNullableFloat64 {
return frame, nil
}
newField := data.NewFieldFromFieldType(data.FieldTypeNullableFloat64, origin.Len())
newField.Name = origin.Name
newField.Labels = origin.Labels
for i := 0; i < origin.Len(); i++ {
v, err := origin.NullableFloatAt(i)
if err != nil {
return frame, err
}
newField.Set(i, v)
}
frame.Fields[Index] = newField
return frame, nil
}
// epochPrecisionToMS converts epoch precision to millisecond, if needed.
// Only seconds to milliseconds supported right now
func epochPrecisionToMS(value float64) float64 {
s := strconv.FormatFloat(value, 'e', -1, 64)
if strings.HasSuffix(s, "e+09") {
return value * float64(1e3)
}
if strings.HasSuffix(s, "e+18") {
return value / float64(time.Millisecond)
}
return value
}
func isDownstreamError(err error) bool {
if backend.IsDownstreamError(err) {
return true
}
resultProcessingDownstreamErrors := []error{
data.ErrorInputFieldsWithoutRows,
data.ErrorSeriesUnsorted,
data.ErrorNullTimeValues,
}
for _, e := range resultProcessingDownstreamErrors {
if errors.Is(err, e) {
return true
}
}
return false
}
@@ -1,681 +0,0 @@
package pgx
import (
"fmt"
"net"
"testing"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgtype"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
)
func Pointer[T any](v T) *T { return &v }
func TestSQLEngine(t *testing.T) {
dt := time.Date(2018, 3, 14, 21, 20, 6, int(527345*time.Microsecond), time.UTC)
t.Run("Handle interpolating $__interval and $__interval_ms", func(t *testing.T) {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := backend.TimeRange{From: from, To: to}
text := "$__interval $__timeGroupAlias(time,$__interval) $__interval_ms"
t.Run("interpolate 10 minutes $__interval", func(t *testing.T) {
query := backend.DataQuery{JSON: []byte("{}"), MaxDataPoints: 1500, Interval: time.Minute * 10}
sql := Interpolate(query, timeRange, "", text)
require.Equal(t, "10m $__timeGroupAlias(time,10m) 600000", sql)
})
t.Run("interpolate 4seconds $__interval", func(t *testing.T) {
query := backend.DataQuery{JSON: []byte("{}"), MaxDataPoints: 1500, Interval: time.Second * 4}
sql := Interpolate(query, timeRange, "", text)
require.Equal(t, "4s $__timeGroupAlias(time,4s) 4000", sql)
})
t.Run("interpolate 200 milliseconds $__interval", func(t *testing.T) {
query := backend.DataQuery{JSON: []byte("{}"), MaxDataPoints: 1500, Interval: time.Millisecond * 200}
sql := Interpolate(query, timeRange, "", text)
require.Equal(t, "200ms $__timeGroupAlias(time,200ms) 200", sql)
})
})
t.Run("Given a time range between 2018-04-12 00:00 and 2018-04-12 00:05", func(t *testing.T) {
from := time.Date(2018, 4, 12, 18, 0, 0, 0, time.UTC)
to := from.Add(5 * time.Minute)
timeRange := backend.TimeRange{From: from, To: to}
query := backend.DataQuery{JSON: []byte("{}"), MaxDataPoints: 1500, Interval: time.Second * 60}
t.Run("interpolate __unixEpochFrom function", func(t *testing.T) {
sql := Interpolate(query, timeRange, "", "select $__unixEpochFrom()")
require.Equal(t, fmt.Sprintf("select %d", from.Unix()), sql)
})
t.Run("interpolate __unixEpochTo function", func(t *testing.T) {
sql := Interpolate(query, timeRange, "", "select $__unixEpochTo()")
require.Equal(t, fmt.Sprintf("select %d", to.Unix()), sql)
})
})
t.Run("Given row values with int64 as time columns", func(t *testing.T) {
tSeconds := dt.Unix()
tMilliseconds := dt.UnixNano() / 1e6
tNanoSeconds := dt.UnixNano()
var nilPointer *int64
originFrame := data.NewFrame("",
data.NewField("time1", nil, []int64{
tSeconds,
}),
data.NewField("time2", nil, []*int64{
Pointer(tSeconds),
}),
data.NewField("time3", nil, []int64{
tMilliseconds,
}),
data.NewField("time4", nil, []*int64{
Pointer(tMilliseconds),
}),
data.NewField("time5", nil, []int64{
tNanoSeconds,
}),
data.NewField("time6", nil, []*int64{
Pointer(tNanoSeconds),
}),
data.NewField("time7", nil, []*int64{
nilPointer,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, dt.Unix(), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[2].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[3].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[4].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[5].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[6].At(0))
})
t.Run("Given row values with uint64 as time columns", func(t *testing.T) {
tSeconds := uint64(dt.Unix())
tMilliseconds := uint64(dt.UnixNano() / 1e6)
tNanoSeconds := uint64(dt.UnixNano())
var nilPointer *uint64
originFrame := data.NewFrame("",
data.NewField("time1", nil, []uint64{
tSeconds,
}),
data.NewField("time2", nil, []*uint64{
Pointer(tSeconds),
}),
data.NewField("time3", nil, []uint64{
tMilliseconds,
}),
data.NewField("time4", nil, []*uint64{
Pointer(tMilliseconds),
}),
data.NewField("time5", nil, []uint64{
tNanoSeconds,
}),
data.NewField("time6", nil, []*uint64{
Pointer(tNanoSeconds),
}),
data.NewField("time7", nil, []*uint64{
nilPointer,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, dt.Unix(), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[2].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[3].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[4].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[5].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[6].At(0))
})
t.Run("Given row values with int32 as time columns", func(t *testing.T) {
tSeconds := int32(dt.Unix())
var nilInt *int32
originFrame := data.NewFrame("",
data.NewField("time1", nil, []int32{
tSeconds,
}),
data.NewField("time2", nil, []*int32{
Pointer(tSeconds),
}),
data.NewField("time7", nil, []*int32{
nilInt,
}),
)
for i := 0; i < 3; i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, dt.Unix(), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[2].At(0))
})
t.Run("Given row values with uint32 as time columns", func(t *testing.T) {
tSeconds := uint32(dt.Unix())
var nilInt *uint32
originFrame := data.NewFrame("",
data.NewField("time1", nil, []uint32{
tSeconds,
}),
data.NewField("time2", nil, []*uint32{
Pointer(tSeconds),
}),
data.NewField("time7", nil, []*uint32{
nilInt,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, dt.Unix(), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[2].At(0))
})
t.Run("Given row values with float64 as time columns", func(t *testing.T) {
tSeconds := float64(dt.UnixNano()) / float64(time.Second)
tMilliseconds := float64(dt.UnixNano()) / float64(time.Millisecond)
tNanoSeconds := float64(dt.UnixNano())
var nilPointer *float64
originFrame := data.NewFrame("",
data.NewField("time1", nil, []float64{
tSeconds,
}),
data.NewField("time2", nil, []*float64{
Pointer(tSeconds),
}),
data.NewField("time3", nil, []float64{
tMilliseconds,
}),
data.NewField("time4", nil, []*float64{
Pointer(tMilliseconds),
}),
data.NewField("time5", nil, []float64{
tNanoSeconds,
}),
data.NewField("time6", nil, []*float64{
Pointer(tNanoSeconds),
}),
data.NewField("time7", nil, []*float64{
nilPointer,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, dt.Unix(), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[2].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[3].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[4].At(0).(*time.Time)).Unix())
require.Equal(t, dt.Unix(), (*originFrame.Fields[5].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[6].At(0))
})
t.Run("Given row values with float32 as time columns", func(t *testing.T) {
tSeconds := float32(dt.Unix())
var nilInt *float32
originFrame := data.NewFrame("",
data.NewField("time1", nil, []float32{
tSeconds,
}),
data.NewField("time2", nil, []*float32{
Pointer(tSeconds),
}),
data.NewField("time7", nil, []*float32{
nilInt,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
err := convertSQLTimeColumnToEpochMS(originFrame, i)
require.NoError(t, err)
}
require.Equal(t, int64(tSeconds), (*originFrame.Fields[0].At(0).(*time.Time)).Unix())
require.Equal(t, int64(tSeconds), (*originFrame.Fields[1].At(0).(*time.Time)).Unix())
require.Nil(t, originFrame.Fields[2].At(0))
})
t.Run("Given row with value columns, would be converted to float64", func(t *testing.T) {
originFrame := data.NewFrame("",
data.NewField("value1", nil, []int64{
int64(1),
}),
data.NewField("value2", nil, []*int64{
Pointer(int64(1)),
}),
data.NewField("value3", nil, []int32{
int32(1),
}),
data.NewField("value4", nil, []*int32{
Pointer(int32(1)),
}),
data.NewField("value5", nil, []int16{
int16(1),
}),
data.NewField("value6", nil, []*int16{
Pointer(int16(1)),
}),
data.NewField("value7", nil, []int8{
int8(1),
}),
data.NewField("value8", nil, []*int8{
Pointer(int8(1)),
}),
data.NewField("value9", nil, []float64{
float64(1),
}),
data.NewField("value10", nil, []*float64{
Pointer(1.0),
}),
data.NewField("value11", nil, []float32{
float32(1),
}),
data.NewField("value12", nil, []*float32{
Pointer(float32(1)),
}),
data.NewField("value13", nil, []uint64{
uint64(1),
}),
data.NewField("value14", nil, []*uint64{
Pointer(uint64(1)),
}),
data.NewField("value15", nil, []uint32{
uint32(1),
}),
data.NewField("value16", nil, []*uint32{
Pointer(uint32(1)),
}),
data.NewField("value17", nil, []uint16{
uint16(1),
}),
data.NewField("value18", nil, []*uint16{
Pointer(uint16(1)),
}),
data.NewField("value19", nil, []uint8{
uint8(1),
}),
data.NewField("value20", nil, []*uint8{
Pointer(uint8(1)),
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
_, err := convertSQLValueColumnToFloat(originFrame, i)
require.NoError(t, err)
if i == 8 {
require.Equal(t, float64(1), originFrame.Fields[i].At(0).(float64))
} else {
require.NotNil(t, originFrame.Fields[i].At(0).(*float64))
require.Equal(t, float64(1), *originFrame.Fields[i].At(0).(*float64))
}
}
})
t.Run("Given row with nil value columns", func(t *testing.T) {
var int64NilPointer *int64
var int32NilPointer *int32
var int16NilPointer *int16
var int8NilPointer *int8
var float64NilPointer *float64
var float32NilPointer *float32
var uint64NilPointer *uint64
var uint32NilPointer *uint32
var uint16NilPointer *uint16
var uint8NilPointer *uint8
originFrame := data.NewFrame("",
data.NewField("value1", nil, []*int64{
int64NilPointer,
}),
data.NewField("value2", nil, []*int32{
int32NilPointer,
}),
data.NewField("value3", nil, []*int16{
int16NilPointer,
}),
data.NewField("value4", nil, []*int8{
int8NilPointer,
}),
data.NewField("value5", nil, []*float64{
float64NilPointer,
}),
data.NewField("value6", nil, []*float32{
float32NilPointer,
}),
data.NewField("value7", nil, []*uint64{
uint64NilPointer,
}),
data.NewField("value8", nil, []*uint32{
uint32NilPointer,
}),
data.NewField("value9", nil, []*uint16{
uint16NilPointer,
}),
data.NewField("value10", nil, []*uint8{
uint8NilPointer,
}),
)
for i := 0; i < len(originFrame.Fields); i++ {
t.Run("", func(t *testing.T) {
_, err := convertSQLValueColumnToFloat(originFrame, i)
require.NoError(t, err)
require.Nil(t, originFrame.Fields[i].At(0))
})
}
})
t.Run("Should not return raw connection errors", func(t *testing.T) {
err := net.OpError{Op: "Dial", Err: fmt.Errorf("inner-error")}
transformer := &testQueryResultTransformer{}
dp := DataSourceHandler{
log: backend.NewLoggerWith("logger", "test"),
queryResultTransformer: transformer,
}
resultErr := dp.TransformQueryError(dp.log, &err)
assert.False(t, transformer.transformQueryErrorWasCalled)
errorText := resultErr.Error()
assert.NotEqual(t, err, resultErr)
assert.NotContains(t, errorText, "inner-error")
assert.Contains(t, errorText, "failed to connect to server")
})
t.Run("Should return non-connection errors unmodified", func(t *testing.T) {
err := fmt.Errorf("normal error")
transformer := &testQueryResultTransformer{}
dp := DataSourceHandler{
log: backend.NewLoggerWith("logger", "test"),
queryResultTransformer: transformer,
}
resultErr := dp.TransformQueryError(dp.log, err)
assert.True(t, transformer.transformQueryErrorWasCalled)
assert.Equal(t, err, resultErr)
assert.ErrorIs(t, err, resultErr)
})
}
func TestConvertResultsToFrame(t *testing.T) {
// Import the pgx packages needed for testing
// These imports are included in the main file but need to be accessible for tests
t.Run("convertResultsToFrame with single result", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
{Name: "value", DataTypeOID: pgtype.Float8OID},
}
// Create mock result data
mockRows := [][][]byte{
{[]byte("1"), []byte("test1"), []byte("10.5")},
{[]byte("2"), []byte("test2"), []byte("20.7")},
}
// Create mock result
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 3, len(frame.Fields))
require.Equal(t, 2, frame.Rows())
// Verify field names
require.Equal(t, "id", frame.Fields[0].Name)
require.Equal(t, "name", frame.Fields[1].Name)
require.Equal(t, "value", frame.Fields[2].Name)
})
t.Run("convertResultsToFrame with multiple compatible results", func(t *testing.T) {
// Create mock field descriptions (same structure for both results)
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
}
// Create first result
mockRows1 := [][][]byte{
{[]byte("1"), []byte("test1")},
{[]byte("2"), []byte("test2")},
}
result1 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows1,
}
result1.CommandTag = pgconn.NewCommandTag("SELECT 2")
// Create second result with same structure
mockRows2 := [][][]byte{
{[]byte("3"), []byte("test3")},
{[]byte("4"), []byte("test4")},
}
result2 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows2,
}
result2.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result1, result2}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 2, len(frame.Fields))
require.Equal(t, 4, frame.Rows()) // Should have rows from both results
// Verify field names
require.Equal(t, "id", frame.Fields[0].Name)
require.Equal(t, "name", frame.Fields[1].Name)
})
t.Run("convertResultsToFrame with row limit", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
// Create mock result data with 3 rows
mockRows := [][][]byte{
{[]byte("1")},
{[]byte("2")},
{[]byte("3")},
}
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 3")
results := []*pgconn.Result{result}
// Set row limit to 2
frame, err := convertResultsToFrame(results, 2)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 2, frame.Rows()) // Should be limited to 2 rows
// Should have a notice about the limit
require.NotNil(t, frame.Meta)
require.Len(t, frame.Meta.Notices, 1)
require.Contains(t, frame.Meta.Notices[0].Text, "Results have been limited to 2")
})
t.Run("convertResultsToFrame with mixed SELECT and non-SELECT results", func(t *testing.T) {
// Create a non-SELECT result (should be skipped)
nonSelectResult := &pgconn.Result{}
nonSelectResult.CommandTag = pgconn.NewCommandTag("UPDATE 1")
// Create a SELECT result
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
mockRows := [][][]byte{
{[]byte("1")},
}
selectResult := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
selectResult.CommandTag = pgconn.NewCommandTag("SELECT 1")
results := []*pgconn.Result{nonSelectResult, selectResult}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 1, frame.Rows())
})
t.Run("convertResultsToFrame with no SELECT results", func(t *testing.T) {
// Create only non-SELECT results
result1 := &pgconn.Result{}
result1.CommandTag = pgconn.NewCommandTag("UPDATE 1")
result2 := &pgconn.Result{}
result2.CommandTag = pgconn.NewCommandTag("INSERT 1")
results := []*pgconn.Result{result1, result2}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 0, len(frame.Fields))
require.Equal(t, 0, frame.Rows())
})
t.Run("convertResultsToFrame with multiple results and row limit per result", func(t *testing.T) {
// Create mock field descriptions (same structure for both results)
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
// Create first result with 3 rows
mockRows1 := [][][]byte{
{[]byte("1")},
{[]byte("2")},
{[]byte("3")},
}
result1 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows1,
}
result1.CommandTag = pgconn.NewCommandTag("SELECT 3")
// Create second result with 3 rows
mockRows2 := [][][]byte{
{[]byte("4")},
{[]byte("5")},
{[]byte("6")},
}
result2 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows2,
}
result2.CommandTag = pgconn.NewCommandTag("SELECT 3")
results := []*pgconn.Result{result1, result2}
// Set row limit to 2 (should limit each result to 2 rows)
frame, err := convertResultsToFrame(results, 2)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 4, frame.Rows()) // 2 rows from each result
// Should have notices about the limit from both results
require.NotNil(t, frame.Meta)
require.Len(t, frame.Meta.Notices, 2)
require.Contains(t, frame.Meta.Notices[0].Text, "Results have been limited to 2")
require.Contains(t, frame.Meta.Notices[1].Text, "Results have been limited to 2")
})
t.Run("convertResultsToFrame handles null values correctly", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
}
// Create mock result data with null values
mockRows := [][][]byte{
{[]byte("1"), nil}, // null name
{nil, []byte("test2")}, // null id
}
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 2, len(frame.Fields))
require.Equal(t, 2, frame.Rows())
// Check that null values are handled correctly
// The exact representation depends on the field type, but should not panic
require.NotPanics(t, func() {
frame.Fields[0].At(1) // null id
frame.Fields[1].At(0) // null name
})
})
}
type testQueryResultTransformer struct {
transformQueryErrorWasCalled bool
}
func (t *testQueryResultTransformer) TransformQueryError(_ log.Logger, err error) error {
t.transformQueryErrorWasCalled = true
return err
}
func (t *testQueryResultTransformer) GetConverterList() []sqlutil.StringConverter {
return nil
}
@@ -1,151 +0,0 @@
package postgres
import (
"fmt"
"os"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
)
var validateCertFuncPgx = validateCertFilePathsPgx
type pgxTlsManager struct {
logger log.Logger
}
func newPgxTlsManager(logger log.Logger) *pgxTlsManager {
return &pgxTlsManager{
logger: logger,
}
}
// getTLSSettings retrieves TLS settings and handles certificate file creation if needed.
func (m *pgxTlsManager) getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error) {
tlsConfig := tlsSettings{
Mode: dsInfo.JsonData.Mode,
}
if tlsConfig.Mode == "disable" {
m.logger.Debug("Postgres TLS/SSL is disabled")
return tlsConfig, nil
}
tlsConfig.ConfigurationMethod = dsInfo.JsonData.ConfigurationMethod
tlsConfig.RootCertFile = dsInfo.JsonData.RootCertFile
tlsConfig.CertFile = dsInfo.JsonData.CertFile
tlsConfig.CertKeyFile = dsInfo.JsonData.CertKeyFile
if tlsConfig.ConfigurationMethod == "file-content" {
if err := m.createCertFiles(dsInfo, &tlsConfig); err != nil {
return tlsConfig, fmt.Errorf("failed to create TLS certificate files: %w", err)
}
} else {
if err := validateCertFuncPgx(tlsConfig.RootCertFile, tlsConfig.CertFile, tlsConfig.CertKeyFile); err != nil {
return tlsConfig, fmt.Errorf("invalid TLS certificate file paths: %w", err)
}
}
return tlsConfig, nil
}
// createCertFiles writes certificate files to temporary locations.
func (m *pgxTlsManager) createCertFiles(dsInfo sqleng.DataSourceInfo, tlsConfig *tlsSettings) error {
m.logger.Debug("Writing TLS certificate files to temporary locations")
var err error
if tlsConfig.RootCertFile, err = m.writeCertFile("root-*.crt", dsInfo.DecryptedSecureJSONData["tlsCACert"]); err != nil {
return err
}
if tlsConfig.CertFile, err = m.writeCertFile("client-*.crt", dsInfo.DecryptedSecureJSONData["tlsClientCert"]); err != nil {
return err
}
if tlsConfig.CertKeyFile, err = m.writeCertFile("client-*.key", dsInfo.DecryptedSecureJSONData["tlsClientKey"]); err != nil {
return err
}
return nil
}
// writeCertFile writes a single certificate file to a temporary location.
func (m *pgxTlsManager) writeCertFile(pattern, content string) (string, error) {
if content == "" {
return "", nil
}
m.logger.Debug("Writing certificate file", "pattern", pattern)
file, err := os.CreateTemp("", pattern)
if err != nil {
return "", fmt.Errorf("failed to create temporary file: %w", err)
}
defer func() {
if err := file.Close(); err != nil {
m.logger.Error("Failed to close file", "error", err)
}
}()
if _, err := file.WriteString(content); err != nil {
return "", fmt.Errorf("failed to write to temporary file: %w", err)
}
return file.Name(), nil
}
// cleanupCertFiles removes temporary certificate files.
func (m *pgxTlsManager) cleanupCertFiles(tlsConfig tlsSettings) {
// Only clean up if the configuration method is "file-content"
if tlsConfig.ConfigurationMethod != "file-content" {
m.logger.Debug("Skipping cleanup of TLS certificate files")
return
}
m.logger.Debug("Cleaning up TLS certificate files")
files := []struct {
path string
name string
}{
{tlsConfig.RootCertFile, "root certificate"},
{tlsConfig.CertFile, "client certificate"},
{tlsConfig.CertKeyFile, "client key"},
}
for _, file := range files {
if file.path == "" {
continue
}
if err := os.Remove(file.path); err != nil {
m.logger.Error("Failed to remove file", "type", file.name, "path", file.path, "error", err)
} else {
m.logger.Debug("Successfully removed file", "type", file.name, "path", file.path)
}
}
}
// validateCertFilePaths validates the existence of configured certificate file paths.
func validateCertFilePathsPgx(rootCert, clientCert, clientKey string) error {
for _, path := range []string{rootCert, clientCert, clientKey} {
if path == "" {
continue
}
exists, err := fileExistsPgx(path)
if err != nil {
return fmt.Errorf("error checking file existence: %w", err)
}
if !exists {
return sqleng.ErrCertFileNotExist
}
}
return nil
}
// fileExists checks if a file exists at the given path.
func fileExistsPgx(path string) (bool, error) {
_, err := os.Stat(path)
if err != nil {
if os.IsNotExist(err) {
return false, nil
}
return false, err
}
return true, nil
}
@@ -1,245 +0,0 @@
package postgres
import (
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// Test getTLSSettings.
func TestPgxGetTLSSettings(t *testing.T) {
mockValidateCertFilePathsPgx()
t.Cleanup(resetValidateCertFilePathsPgx)
updatedTime := time.Now()
testCases := []struct {
desc string
expErr string
jsonData sqleng.JsonData
secureJSONData map[string]string
uid string
tlsSettings tlsSettings
updated time.Time
}{
{
desc: "Custom TLS authentication disabled",
updated: updatedTime,
jsonData: sqleng.JsonData{
Mode: "disable",
RootCertFile: "i/am/coding/ca.crt",
CertFile: "i/am/coding/client.crt",
CertKeyFile: "i/am/coding/client.key",
ConfigurationMethod: "file-path",
},
tlsSettings: tlsSettings{Mode: "disable"},
},
{
desc: "Custom TLS authentication with file path",
updated: updatedTime.Add(time.Minute),
jsonData: sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-path",
RootCertFile: "i/am/coding/ca.crt",
CertFile: "i/am/coding/client.crt",
CertKeyFile: "i/am/coding/client.key",
},
tlsSettings: tlsSettings{
Mode: "verify-full",
ConfigurationMethod: "file-path",
RootCertFile: "i/am/coding/ca.crt",
CertFile: "i/am/coding/client.crt",
CertKeyFile: "i/am/coding/client.key",
},
},
}
for _, tt := range testCases {
t.Run(tt.desc, func(t *testing.T) {
var settings tlsSettings
var err error
mng := pgxTlsManager{
logger: backend.NewLoggerWith("logger", "tsdb.postgres"),
}
ds := sqleng.DataSourceInfo{
JsonData: tt.jsonData,
DecryptedSecureJSONData: tt.secureJSONData,
UID: tt.uid,
Updated: tt.updated,
}
settings, err = mng.getTLSSettings(ds)
if tt.expErr == "" {
require.NoError(t, err, tt.desc)
assert.Equal(t, tt.tlsSettings, settings)
} else {
require.Error(t, err, tt.desc)
assert.True(t, strings.HasPrefix(err.Error(), tt.expErr),
fmt.Sprintf("%s: %q doesn't start with %q", tt.desc, err, tt.expErr))
}
})
}
}
func mockValidateCertFilePathsPgx() {
validateCertFuncPgx = func(rootCert, clientCert, clientKey string) error {
return nil
}
}
func resetValidateCertFilePathsPgx() {
validateCertFuncPgx = validateCertFilePathsPgx
}
func TestTLSManager_GetTLSSettings(t *testing.T) {
logger := log.New()
tlsManager := newPgxTlsManager(logger)
dsInfo := sqleng.DataSourceInfo{
JsonData: sqleng.JsonData{
Mode: "require",
ConfigurationMethod: "file-content",
},
DecryptedSecureJSONData: map[string]string{
"tlsCACert": "root-cert-content",
"tlsClientCert": "client-cert-content",
"tlsClientKey": "client-key-content",
},
}
tlsConfig, err := tlsManager.getTLSSettings(dsInfo)
require.NoError(t, err)
assert.Equal(t, "require", tlsConfig.Mode)
assert.NotEmpty(t, tlsConfig.RootCertFile)
assert.NotEmpty(t, tlsConfig.CertFile)
assert.NotEmpty(t, tlsConfig.CertKeyFile)
// Cleanup temporary files
tlsManager.cleanupCertFiles(tlsConfig)
assert.NoFileExists(t, tlsConfig.RootCertFile)
assert.NoFileExists(t, tlsConfig.CertFile)
assert.NoFileExists(t, tlsConfig.CertKeyFile)
}
func TestTLSManager_CleanupCertFiles_FilePath(t *testing.T) {
logger := log.New()
tlsManager := newPgxTlsManager(logger)
// Create temporary files for testing
rootCertFile, err := tlsManager.writeCertFile("root-*.crt", "root-cert-content")
require.NoError(t, err)
clientCertFile, err := tlsManager.writeCertFile("client-*.crt", "client-cert-content")
require.NoError(t, err)
clientKeyFile, err := tlsManager.writeCertFile("client-*.key", "client-key-content")
require.NoError(t, err)
// Simulate a configuration where the method is "file-path"
tlsConfig := tlsSettings{
ConfigurationMethod: "file-path",
RootCertFile: rootCertFile,
CertFile: clientCertFile,
CertKeyFile: clientKeyFile,
}
// Call cleanupCertFiles
tlsManager.cleanupCertFiles(tlsConfig)
// Verify the files are NOT deleted
assert.FileExists(t, rootCertFile, "Root certificate file should not be deleted")
assert.FileExists(t, clientCertFile, "Client certificate file should not be deleted")
assert.FileExists(t, clientKeyFile, "Client key file should not be deleted")
// Cleanup the files manually
err = os.Remove(rootCertFile)
require.NoError(t, err)
err = os.Remove(clientCertFile)
require.NoError(t, err)
err = os.Remove(clientKeyFile)
require.NoError(t, err)
}
func TestTLSManager_CreateCertFiles(t *testing.T) {
logger := log.New()
tlsManager := newPgxTlsManager(logger)
dsInfo := sqleng.DataSourceInfo{
DecryptedSecureJSONData: map[string]string{
"tlsCACert": "root-cert-content",
"tlsClientCert": "client-cert-content",
"tlsClientKey": "client-key-content",
},
}
tlsConfig := tlsSettings{
ConfigurationMethod: "file-content",
}
err := tlsManager.createCertFiles(dsInfo, &tlsConfig)
require.NoError(t, err)
assert.FileExists(t, tlsConfig.RootCertFile)
assert.FileExists(t, tlsConfig.CertFile)
assert.FileExists(t, tlsConfig.CertKeyFile)
// Cleanup temporary files
tlsManager.cleanupCertFiles(tlsConfig)
assert.NoFileExists(t, tlsConfig.RootCertFile)
assert.NoFileExists(t, tlsConfig.CertFile)
assert.NoFileExists(t, tlsConfig.CertKeyFile)
}
func TestTLSManager_WriteCertFile(t *testing.T) {
logger := log.New()
tlsManager := newPgxTlsManager(logger)
// Test writing a valid certificate file
filePath, err := tlsManager.writeCertFile("test-*.crt", "test-cert-content")
require.NoError(t, err)
assert.FileExists(t, filePath)
content, err := os.ReadFile(filepath.Clean(filePath))
require.NoError(t, err)
assert.Equal(t, "test-cert-content", string(content))
// Cleanup the file
err = os.Remove(filePath)
require.NoError(t, err)
assert.NoFileExists(t, filePath)
}
func TestTLSManager_CleanupCertFiles(t *testing.T) {
logger := log.New()
tlsManager := newPgxTlsManager(logger)
// Create temporary files for testing
rootCertFile, err := tlsManager.writeCertFile("root-*.crt", "root-cert-content")
require.NoError(t, err)
clientCertFile, err := tlsManager.writeCertFile("client-*.crt", "client-cert-content")
require.NoError(t, err)
clientKeyFile, err := tlsManager.writeCertFile("client-*.key", "client-key-content")
require.NoError(t, err)
tlsConfig := tlsSettings{
ConfigurationMethod: "file-content",
RootCertFile: rootCertFile,
CertFile: clientCertFile,
CertKeyFile: clientKeyFile,
}
// Cleanup the files
tlsManager.cleanupCertFiles(tlsConfig)
// Verify the files are deleted
assert.NoFileExists(t, rootCertFile)
assert.NoFileExists(t, clientCertFile)
assert.NoFileExists(t, clientKeyFile)
}
@@ -2,7 +2,6 @@ package postgres
import (
"context"
"database/sql"
"encoding/json"
"fmt"
"reflect"
@@ -16,63 +15,12 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/lib/pq"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
sqlengpgx "github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/pgx"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
)
func newPostgres(ctx context.Context, userFacingDefaultError string, rowLimit int64, dsInfo sqleng.DataSourceInfo, cnnstr string, logger log.Logger, settings backend.DataSourceInstanceSettings) (*sql.DB, *sqleng.DataSourceHandler, error) {
connector, err := pq.NewConnector(cnnstr)
if err != nil {
logger.Error("postgres connector creation failed", "error", err)
return nil, nil, fmt.Errorf("postgres connector creation failed")
}
proxyClient, err := settings.ProxyClient(ctx)
if err != nil {
logger.Error("postgres proxy creation failed", "error", err)
return nil, nil, fmt.Errorf("postgres proxy creation failed")
}
if proxyClient.SecureSocksProxyEnabled() {
dialer, err := proxyClient.NewSecureSocksProxyContextDialer()
if err != nil {
logger.Error("postgres proxy creation failed", "error", err)
return nil, nil, fmt.Errorf("postgres proxy creation failed")
}
postgresDialer := newPostgresProxyDialer(dialer)
// update the postgres dialer with the proxy dialer
connector.Dialer(postgresDialer)
}
config := sqleng.DataPluginConfiguration{
DSInfo: dsInfo,
MetricColumnTypes: []string{"UNKNOWN", "TEXT", "VARCHAR", "CHAR"},
RowLimit: rowLimit,
}
queryResultTransformer := postgresQueryResultTransformer{}
db := sql.OpenDB(connector)
db.SetMaxOpenConns(config.DSInfo.JsonData.MaxOpenConns)
db.SetMaxIdleConns(config.DSInfo.JsonData.MaxIdleConns)
db.SetConnMaxLifetime(time.Duration(config.DSInfo.JsonData.ConnMaxLifetime) * time.Second)
handler, err := sqleng.NewQueryDataHandler(userFacingDefaultError, db, config, &queryResultTransformer, newPostgresMacroEngine(dsInfo.JsonData.Timescaledb),
logger)
if err != nil {
logger.Error("Failed connecting to Postgres", "err", err)
return nil, nil, err
}
logger.Debug("Successfully connected to Postgres")
return db, handler, nil
}
func newPostgresPGX(ctx context.Context, userFacingDefaultError string, rowLimit int64, dsInfo sqleng.DataSourceInfo, cnnstr string, logger log.Logger, settings backend.DataSourceInstanceSettings) (*pgxpool.Pool, *sqlengpgx.DataSourceHandler, error) {
func newPostgres(ctx context.Context, userFacingDefaultError string, rowLimit int64, dsInfo sqleng.DataSourceInfo, cnnstr string, logger log.Logger, settings backend.DataSourceInstanceSettings) (*pgxpool.Pool, *sqleng.DataSourceHandler, error) {
pgxConf, err := pgxpool.ParseConfig(cnnstr)
if err != nil {
logger.Error("postgres config creation failed", "error", err)
@@ -92,7 +40,7 @@ func newPostgresPGX(ctx context.Context, userFacingDefaultError string, rowLimit
return nil, nil, fmt.Errorf("postgres proxy creation failed")
}
pgxConf.ConnConfig.DialFunc = newPgxDialFunc(dialer)
pgxConf.ConnConfig.DialFunc = newDialFunc(dialer)
}
// by default pgx resolves hostnames to ip addresses. we must avoid this.
@@ -101,7 +49,7 @@ func newPostgresPGX(ctx context.Context, userFacingDefaultError string, rowLimit
return []string{host}, nil
}
config := sqlengpgx.DataPluginConfiguration{
config := sqleng.DataPluginConfiguration{
DSInfo: dsInfo,
MetricColumnTypes: []string{"unknown", "text", "varchar", "char", "bpchar"},
RowLimit: rowLimit,
@@ -117,7 +65,7 @@ func newPostgresPGX(ctx context.Context, userFacingDefaultError string, rowLimit
return nil, nil, err
}
handler, err := sqlengpgx.NewQueryDataHandler(userFacingDefaultError, p, config, &queryResultTransformer, newPostgresMacroEngine(dsInfo.JsonData.Timescaledb),
handler, err := sqleng.NewQueryDataHandler(userFacingDefaultError, p, config, &queryResultTransformer, newPostgresMacroEngine(dsInfo.JsonData.Timescaledb),
logger)
if err != nil {
logger.Error("Failed connecting to Postgres", "err", err)
@@ -128,7 +76,7 @@ func newPostgresPGX(ctx context.Context, userFacingDefaultError string, rowLimit
return p, handler, nil
}
func NewInstanceSettings(logger log.Logger, dataPath string) datasource.InstanceFactoryFunc {
func NewInstanceSettings(logger log.Logger) datasource.InstanceFactoryFunc {
return func(ctx context.Context, settings backend.DataSourceInstanceSettings) (instancemgmt.Instance, error) {
cfg := backend.GrafanaConfigFromContext(ctx)
sqlCfg, err := cfg.SQL()
@@ -171,48 +119,26 @@ func NewInstanceSettings(logger log.Logger, dataPath string) datasource.Instance
return nil, err
}
usePGX := cfg.FeatureToggles().IsEnabled("postgresDSUsePGX")
if usePGX {
pgxlogger := logger.FromContext(ctx).With("driver", "pgx")
pgxTlsManager := newPgxTlsManager(pgxlogger)
pgxTlsSettings, err := pgxTlsManager.getTLSSettings(dsInfo)
if err != nil {
return "", err
}
// Ensure cleanupCertFiles is called after the connection is opened
defer pgxTlsManager.cleanupCertFiles(pgxTlsSettings)
cnnstr, err := generateConnectionString(dsInfo, pgxTlsSettings, usePGX, pgxlogger)
if err != nil {
return "", err
}
_, handler, err := newPostgresPGX(ctx, userFacingDefaultError, sqlCfg.RowLimit, dsInfo, cnnstr, pgxlogger, settings)
if err != nil {
pgxlogger.Error("Failed connecting to Postgres", "err", err)
return nil, err
}
pgxlogger.Debug("Successfully connected to Postgres")
return handler, nil
} else {
pqlogger := logger.FromContext(ctx).With("driver", "libpq")
tlsManager := newTLSManager(pqlogger, dataPath)
tlsSettings, err := tlsManager.getTLSSettings(dsInfo)
if err != nil {
return "", err
}
cnnstr, err := generateConnectionString(dsInfo, tlsSettings, usePGX, pqlogger)
if err != nil {
return nil, err
}
_, handler, err := newPostgres(ctx, userFacingDefaultError, sqlCfg.RowLimit, dsInfo, cnnstr, pqlogger, settings)
if err != nil {
pqlogger.Error("Failed connecting to Postgres", "err", err)
return nil, err
}
pqlogger.Debug("Successfully connected to Postgres")
return handler, nil
pgxlogger := logger.FromContext(ctx).With("driver", "pgx")
tlsManager := newTLSManager(pgxlogger)
tlsSettings, err := tlsManager.getTLSSettings(dsInfo)
if err != nil {
return "", err
}
// Ensure cleanupCertFiles is called after the connection is opened
defer tlsManager.cleanupCertFiles(tlsSettings)
cnnstr, err := generateConnectionString(dsInfo, tlsSettings, pgxlogger)
if err != nil {
return "", err
}
_, handler, err := newPostgres(ctx, userFacingDefaultError, sqlCfg.RowLimit, dsInfo, cnnstr, pgxlogger, settings)
if err != nil {
pgxlogger.Error("Failed connecting to Postgres", "err", err)
return nil, err
}
pgxlogger.Debug("Successfully connected to Postgres")
return handler, nil
}
}
@@ -302,7 +228,7 @@ func buildBaseConnectionString(params connectionParams) string {
return connStr
}
func generateConnectionString(dsInfo sqleng.DataSourceInfo, tlsSettings tlsSettings, isPGX bool, logger log.Logger) (string, error) {
func generateConnectionString(dsInfo sqleng.DataSourceInfo, tlsSettings tlsSettings, logger log.Logger) (string, error) {
params, err := parseConnectionParams(dsInfo, logger)
if err != nil {
return "", err
@@ -312,15 +238,6 @@ func generateConnectionString(dsInfo sqleng.DataSourceInfo, tlsSettings tlsSetti
connStr += fmt.Sprintf(" sslmode='%s'", escape(tlsSettings.Mode))
// there is an issue with the lib/pq module, the `verify-ca` tls mode
// does not work correctly. ( see https://github.com/lib/pq/issues/1106 )
// to workaround the problem, if the `verify-ca` mode is chosen,
// we disable sslsni.
if tlsSettings.Mode == "verify-ca" && !isPGX {
logger.Debug("Disabling sslsni for verify-ca mode")
connStr += " sslsni=0"
}
// Attach root certificate if provided
if tlsSettings.RootCertFile != "" {
logger.Debug("Setting server root certificate", "tlsRootCert", tlsSettings.RootCertFile)
@@ -1,196 +0,0 @@
package postgres
import (
"context"
"encoding/json"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"testing"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana-plugin-sdk-go/experimental"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/grafana/grafana/pkg/util/testutil"
)
// These tests require a real postgres database:
// - make devenv sources=postgres_tests
// - either set the env variable GRAFANA_TEST_DB = postgres
// - or set `forceRun := true` below
//
// The tests require a PostgreSQL db named grafanadstest and a user/password grafanatest/grafanatest!
// Use the docker/blocks/postgres_tests/docker-compose.yaml to spin up a
// preconfigured Postgres server suitable for running these tests.
func TestIntegrationPostgresPGXSnapshots(t *testing.T) {
testutil.SkipIntegrationTestInShortMode(t)
// the logic in this function is copied from postgres_tests.go
shouldRunTest := func() bool {
testDbName, present := os.LookupEnv("GRAFANA_TEST_DB")
if present && testDbName == "postgres" {
return true
}
return false
}
if !shouldRunTest() {
t.Skip()
}
getCnnStr := func() string {
host := os.Getenv("POSTGRES_HOST")
if host == "" {
host = "localhost"
}
port := os.Getenv("POSTGRES_PORT")
if port == "" {
port = "5432"
}
return fmt.Sprintf("user=grafanatest password=grafanatest host=%s port=%s dbname=grafanadstest sslmode=disable",
host, port)
}
sqlQueryCommentRe := regexp.MustCompile(`^-- (.+)\n`)
readSqlFile := func(path string) (string, string) {
// the file-path is not coming from the outside,
// it is hardcoded in this file.
//nolint:gosec
sqlBytes, err := os.ReadFile(path)
require.NoError(t, err)
sql := string(sqlBytes)
// first line of the file contains the sql query to run, commented out
match := sqlQueryCommentRe.FindStringSubmatch(sql)
require.Len(t, match, 2)
rawSQL := strings.TrimSpace(match[1])
return rawSQL, sql
}
makeQuery := func(rawSQL string, format string) backend.QueryDataRequest {
queryData := map[string]string{
"rawSql": rawSQL,
"format": format,
}
queryBytes, err := json.Marshal(queryData)
require.NoError(t, err)
return backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
JSON: queryBytes,
RefID: "A",
TimeRange: backend.TimeRange{
From: time.Date(2023, 12, 24, 14, 15, 22, 123456, time.UTC),
To: time.Date(2023, 12, 24, 14, 45, 13, 876543, time.UTC),
},
},
},
}
}
tt := []struct {
name string
format string
}{
{format: "time_series", name: "simple"},
{format: "time_series", name: "no_rows_long"},
{format: "time_series", name: "no_rows_wide"},
{format: "time_series", name: "7x_compat_metric_label"},
{format: "time_series", name: "convert_to_float64"},
{format: "time_series", name: "convert_to_float64_not"},
{format: "time_series", name: "fill_null"},
{format: "time_series", name: "fill_previous"},
{format: "time_series", name: "fill_value"},
{format: "time_series", name: "fill_value_wide"},
{format: "table", name: "simple"},
{format: "table", name: "multi_stat1"},
{format: "table", name: "multi_stat2"},
{format: "table", name: "no_rows"},
{format: "table", name: "types_numeric"},
{format: "table", name: "types_char"},
{format: "table", name: "types_datetime_pgx"},
{format: "table", name: "types_other"},
{format: "table", name: "types_enum"},
{format: "table", name: "types_jsonb"},
{format: "table", name: "timestamp_convert_bigint"},
{format: "table", name: "timestamp_convert_integer"},
{format: "table", name: "timestamp_convert_real"},
{format: "table", name: "timestamp_convert_double"},
{format: "table", name: "time_group_compat_case1"},
{format: "table", name: "time_group_compat_case2"},
}
for _, test := range tt {
require.True(t, test.format == "table" || test.format == "time_series")
t.Run(test.name, func(t *testing.T) {
origInterpolate := sqleng.Interpolate
t.Cleanup(func() {
sqleng.Interpolate = origInterpolate
})
sqleng.Interpolate = func(query backend.DataQuery, timeRange backend.TimeRange, timeInterval string, sql string) string {
return sql
}
jsonData := sqleng.JsonData{
MaxOpenConns: 10,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Timescaledb: false,
Mode: "disable",
ConfigurationMethod: "file-path",
}
dsInfo := sqleng.DataSourceInfo{
JsonData: jsonData,
DecryptedSecureJSONData: map[string]string{},
}
logger := log.New()
cnnstr := getCnnStr()
p, handler, err := newPostgresPGX(context.Background(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
t.Cleanup((func() {
_, err := p.Exec(context.Background(), "DROP TABLE tbl")
require.NoError(t, err)
p.Close()
}))
require.NoError(t, err)
sqlFilePath := filepath.Join("testdata", test.format, test.name+".sql")
goldenFileName := filepath.Join(test.format, test.name+".golden")
rawSQL, sql := readSqlFile(sqlFilePath)
_, err = p.Exec(context.Background(), sql)
require.NoError(t, err)
query := makeQuery(rawSQL, test.format)
result, err := handler.QueryData(context.Background(), &query)
require.Len(t, result.Responses, 1)
response, found := result.Responses["A"]
require.True(t, found)
require.NoError(t, err)
experimental.CheckGoldenJSONResponse(t, "testdata", goldenFileName, &response, updateGoldenFiles)
})
}
}
File diff suppressed because it is too large Load Diff
@@ -7,8 +7,6 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend/datasource"
"github.com/grafana/grafana-plugin-sdk-go/backend/instancemgmt"
"github.com/grafana/grafana/pkg/setting"
sqlengpgx "github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/pgx"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
)
@@ -16,48 +14,30 @@ type Service struct {
im instancemgmt.InstanceManager
}
func ProvideService(cfg *setting.Cfg) *Service {
func ProvideService() *Service {
logger := backend.NewLoggerWith("logger", "tsdb.postgres")
s := &Service{
im: datasource.NewInstanceManager(NewInstanceSettings(logger, cfg.DataPath)),
im: datasource.NewInstanceManager(NewInstanceSettings(logger)),
}
return s
}
// NOTE: do not put any business logic into this method. it's whole job is to forward the call "inside"
func (s *Service) CheckHealth(ctx context.Context, req *backend.CheckHealthRequest) (*backend.CheckHealthResult, error) {
cfg := backend.GrafanaConfigFromContext(ctx)
if cfg.FeatureToggles().IsEnabled("postgresDSUsePGX") {
dsHandler, err := s.getDSInfoPGX(ctx, req.PluginContext)
if err != nil {
return sqlengpgx.ErrToHealthCheckResult(err)
}
return dsHandler.CheckHealth(ctx, req)
} else {
dsHandler, err := s.getDSInfo(ctx, req.PluginContext)
if err != nil {
return sqleng.ErrToHealthCheckResult(err)
}
return dsHandler.CheckHealth(ctx, req)
dsHandler, err := s.getDSInfo(ctx, req.PluginContext)
if err != nil {
return sqleng.ErrToHealthCheckResult(err)
}
return dsHandler.CheckHealth(ctx, req)
}
// NOTE: do not put any business logic into this method. it's whole job is to forward the call "inside"
func (s *Service) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
cfg := backend.GrafanaConfigFromContext(ctx)
if cfg.FeatureToggles().IsEnabled("postgresDSUsePGX") {
dsInfo, err := s.getDSInfoPGX(ctx, req.PluginContext)
if err != nil {
return nil, err
}
return dsInfo.QueryData(ctx, req)
} else {
dsInfo, err := s.getDSInfo(ctx, req.PluginContext)
if err != nil {
return nil, err
}
return dsInfo.QueryData(ctx, req)
dsInfo, err := s.getDSInfo(ctx, req.PluginContext)
if err != nil {
return nil, err
}
return dsInfo.QueryData(ctx, req)
}
func (s *Service) getDSInfo(ctx context.Context, pluginCtx backend.PluginContext) (*sqleng.DataSourceHandler, error) {
@@ -68,12 +48,3 @@ func (s *Service) getDSInfo(ctx context.Context, pluginCtx backend.PluginContext
instance := i.(*sqleng.DataSourceHandler)
return instance, nil
}
func (s *Service) getDSInfoPGX(ctx context.Context, pluginCtx backend.PluginContext) (*sqlengpgx.DataSourceHandler, error) {
i, err := s.im.Get(ctx, pluginCtx)
if err != nil {
return nil, err
}
instance := i.(*sqlengpgx.DataSourceHandler)
return instance, nil
}
@@ -1,7 +1,6 @@
package postgres
import (
"context"
"encoding/json"
"fmt"
"os"
@@ -23,7 +22,7 @@ import (
var updateGoldenFiles = false
// These tests require a real postgres database:
// - make devenv sources=potgres_tests
// - make devenv sources=postgres_tests
// - either set the env variable GRAFANA_TEST_DB = postgres
// - or set `forceRun := true` below
//
@@ -127,6 +126,8 @@ func TestIntegrationPostgresSnapshots(t *testing.T) {
{format: "table", name: "types_char"},
{format: "table", name: "types_datetime"},
{format: "table", name: "types_other"},
{format: "table", name: "types_enum"},
{format: "table", name: "types_jsonb"},
{format: "table", name: "timestamp_convert_bigint"},
{format: "table", name: "timestamp_convert_integer"},
{format: "table", name: "timestamp_convert_real"},
@@ -148,10 +149,11 @@ func TestIntegrationPostgresSnapshots(t *testing.T) {
}
jsonData := sqleng.JsonData{
MaxOpenConns: 0,
MaxOpenConns: 10,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Timescaledb: false,
Mode: "disable",
ConfigurationMethod: "file-path",
}
@@ -164,28 +166,24 @@ func TestIntegrationPostgresSnapshots(t *testing.T) {
cnnstr := getCnnStr()
db, handler, err := newPostgres(context.Background(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
t.Cleanup((func() {
_, err := db.Exec("DROP TABLE tbl")
require.NoError(t, err)
err = db.Close()
require.NoError(t, err)
}))
p, handler, err := newPostgres(t.Context(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
require.NoError(t, err)
t.Cleanup(func() {
p.Close()
})
sqlFilePath := filepath.Join("testdata", test.format, test.name+".sql")
goldenFileName := filepath.Join(test.format, test.name+".golden")
rawSQL, sql := readSqlFile(sqlFilePath)
_, err = db.Exec(sql)
_, err = p.Exec(t.Context(), sql)
require.NoError(t, err)
query := makeQuery(rawSQL, test.format)
result, err := handler.QueryData(context.Background(), &query)
result, err := handler.QueryData(t.Context(), &query)
require.Len(t, result.Responses, 1)
response, found := result.Responses["A"]
require.True(t, found)
@@ -1,7 +1,7 @@
package postgres
import (
"context"
"encoding/json"
"fmt"
"math/rand"
"os"
@@ -15,20 +15,14 @@ import (
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/grafana/grafana/pkg/util/testutil"
_ "github.com/lib/pq"
)
// Test generateConnectionString.
func TestIntegrationGenerateConnectionString(t *testing.T) {
testutil.SkipIntegrationTestInShortMode(t)
cfg := setting.NewCfg()
cfg.DataPath = t.TempDir()
testCases := []struct {
desc string
host string
@@ -58,15 +52,6 @@ func TestIntegrationGenerateConnectionString(t *testing.T) {
tlsSettings: tlsSettings{Mode: "verify-full"},
expConnStr: "user='user' host='host' dbname='database' password='password' sslmode='verify-full'",
},
{
desc: "verify-ca automatically adds disable-sni",
host: "host:1234",
user: "user",
password: "password",
database: "database",
tlsSettings: tlsSettings{Mode: "verify-ca"},
expConnStr: "user='user' host='host' dbname='database' password='password' port=1234 sslmode='verify-ca' sslsni=0",
},
{
desc: "TCP/port host",
host: "host:1234",
@@ -156,8 +141,6 @@ func TestIntegrationGenerateConnectionString(t *testing.T) {
}
for _, tt := range testCases {
t.Run(tt.desc, func(t *testing.T) {
logger := backend.NewLoggerWith("logger", "tsdb.postgres")
ds := sqleng.DataSourceInfo{
URL: tt.host,
User: tt.user,
@@ -165,8 +148,9 @@ func TestIntegrationGenerateConnectionString(t *testing.T) {
Database: tt.database,
UID: tt.uid,
}
logger := backend.NewLoggerWith("logger", "tsdb.postgres")
connStr, err := generateConnectionString(ds, tt.tlsSettings, false, logger)
connStr, err := generateConnectionString(ds, tt.tlsSettings, logger)
if tt.expErr == "" {
require.NoError(t, err, tt.desc)
@@ -207,10 +191,11 @@ func TestIntegrationPostgres(t *testing.T) {
}
jsonData := sqleng.JsonData{
MaxOpenConns: 0,
MaxOpenConns: 10,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Timescaledb: false,
Mode: "disable",
ConfigurationMethod: "file-path",
}
@@ -223,7 +208,7 @@ func TestIntegrationPostgres(t *testing.T) {
cnnstr := postgresTestDBConnString()
db, exe, err := newPostgres(context.Background(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
p, exe, err := newPostgres(t.Context(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
require.NoError(t, err)
@@ -253,10 +238,13 @@ func TestIntegrationPostgres(t *testing.T) {
c14_timetz time with time zone,
time date,
c15_interval interval,
c16_smallint smallint
c16_smallint smallint,
c17_json json,
c18_jsonb jsonb
);
`
_, err := db.Exec(sql)
_, err := p.Exec(t.Context(), sql)
require.NoError(t, err)
sql = `
@@ -264,12 +252,11 @@ func TestIntegrationPostgres(t *testing.T) {
1,2,3,
4.5,6.7,1.1,1.2,
'char10','varchar10','text',
now(),now(),now(),now(),now(),now(),'15m'::interval,
null
now(),now(),now(),now(),now(),now(),'15m'::interval,null,
'{"key1": "value1"}'::json, '{"key2": "value2"}'::jsonb
);
`
_, err = db.Exec(sql)
_, err = p.Exec(t.Context(), sql)
require.NoError(t, err)
t.Run("When doing a table query should map Postgres column types to Go types", func(t *testing.T) {
@@ -284,14 +271,14 @@ func TestIntegrationPostgres(t *testing.T) {
},
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
frames := queryResult.Frames
require.Len(t, frames, 1)
require.Len(t, frames[0].Fields, 18)
require.Len(t, frames[0].Fields, 20)
require.Equal(t, int16(1), *frames[0].Fields[0].At(0).(*int16))
require.Equal(t, int32(2), *frames[0].Fields[1].At(0).(*int32))
@@ -312,14 +299,21 @@ func TestIntegrationPostgres(t *testing.T) {
require.True(t, ok)
_, ok = frames[0].Fields[12].At(0).(*time.Time)
require.True(t, ok)
_, ok = frames[0].Fields[13].At(0).(*time.Time)
_, ok = frames[0].Fields[13].At(0).(*string)
require.True(t, ok)
_, ok = frames[0].Fields[14].At(0).(*time.Time)
_, ok = frames[0].Fields[14].At(0).(*string)
require.True(t, ok)
_, ok = frames[0].Fields[15].At(0).(*time.Time)
require.True(t, ok)
require.Equal(t, "00:15:00", *frames[0].Fields[16].At(0).(*string))
require.Nil(t, frames[0].Fields[17].At(0))
_, ok = frames[0].Fields[18].At(0).(*json.RawMessage)
require.True(t, ok)
require.Equal(t, json.RawMessage(`{"key1": "value1"}`), *frames[0].Fields[18].At(0).(*json.RawMessage))
_, ok = frames[0].Fields[19].At(0).(*json.RawMessage)
require.True(t, ok)
require.Equal(t, json.RawMessage(`{"key2": "value2"}`), *frames[0].Fields[19].At(0).(*json.RawMessage))
})
})
@@ -332,7 +326,7 @@ func TestIntegrationPostgres(t *testing.T) {
)
`
_, err := db.Exec(sql)
_, err := p.Exec(t.Context(), sql)
require.NoError(t, err)
type metric struct {
@@ -359,7 +353,7 @@ func TestIntegrationPostgres(t *testing.T) {
}
for _, m := range series {
_, err := db.Exec(`INSERT INTO metric ("time", value) VALUES ($1, $2)`, m.Time.UTC(), m.Value)
_, err := p.Exec(t.Context(), `INSERT INTO metric ("time", value) VALUES ($1, $2)`, m.Time.UTC(), m.Value)
require.NoError(t, err)
}
@@ -376,7 +370,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -408,6 +402,27 @@ func TestIntegrationPostgres(t *testing.T) {
}
})
t.Run("When doing a query without a format should default to time_series", func(t *testing.T) {
query := &backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
JSON: []byte(`{
"rawSql": "SELECT $__timeGroup(time, '5m') AS time, avg(value) as value FROM metric GROUP BY 1 ORDER BY 1 "
}`),
RefID: "A",
},
},
}
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
frames := queryResult.Frames
require.Len(t, frames, 1)
require.Len(t, frames[0].Fields, 2)
})
t.Run("When doing a metric query using timeGroup and $__interval", func(t *testing.T) {
mockInterpolate := sqleng.Interpolate
sqleng.Interpolate = origInterpolate
@@ -432,7 +447,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
frames := queryResult.Frames
@@ -460,7 +475,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -514,7 +529,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -540,7 +555,7 @@ func TestIntegrationPostgres(t *testing.T) {
}
for _, m := range series {
_, err := db.Exec(`INSERT INTO metric ("time", value) VALUES ($1, $2)`, m.Time.UTC(), m.Value)
_, err := p.Exec(t.Context(), `INSERT INTO metric ("time", value) VALUES ($1, $2)`, m.Time.UTC(), m.Value)
require.NoError(t, err)
}
@@ -561,7 +576,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -596,7 +611,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -624,10 +639,10 @@ func TestIntegrationPostgres(t *testing.T) {
ValueTwo int64
}
_, err := db.Exec("DROP TABLE IF EXISTS metric_values")
_, err := p.Exec(t.Context(), "DROP TABLE IF EXISTS metric_values")
require.NoError(t, err)
_, err = db.Exec(`CREATE TABLE metric_values (
_, err = p.Exec(t.Context(), `CREATE TABLE metric_values (
"time" TIMESTAMP NULL,
"timeInt64" BIGINT NOT NULL, "timeInt64Nullable" BIGINT NULL,
"timeFloat64" DOUBLE PRECISION NOT NULL, "timeFloat64Nullable" DOUBLE PRECISION NULL,
@@ -680,7 +695,7 @@ func TestIntegrationPostgres(t *testing.T) {
// _, err = session.InsertMulti(series)
for _, m := range series {
_, err := db.Exec(`INSERT INTO "metric_values" (
_, err := p.Exec(t.Context(), `INSERT INTO "metric_values" (
time,
"timeInt64", "timeInt64Nullable",
"timeFloat64", "timeFloat64Nullable",
@@ -713,7 +728,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -737,7 +752,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -761,7 +776,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -785,7 +800,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -809,7 +824,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -833,7 +848,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -857,7 +872,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -882,7 +897,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -906,7 +921,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -931,7 +946,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -963,7 +978,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -998,7 +1013,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1017,9 +1032,9 @@ func TestIntegrationPostgres(t *testing.T) {
Tags string
}
_, err := db.Exec("DROP TABLE IF EXISTS event")
_, err := p.Exec(t.Context(), "DROP TABLE IF EXISTS event")
require.NoError(t, err)
_, err = db.Exec(`CREATE TABLE event (time_sec BIGINT NULL, description VARCHAR(255) NULL, tags VARCHAR(255) NULL)`)
_, err = p.Exec(t.Context(), `CREATE TABLE event (time_sec BIGINT NULL, description VARCHAR(255) NULL, tags VARCHAR(255) NULL)`)
require.NoError(t, err)
events := []*event{}
@@ -1037,7 +1052,7 @@ func TestIntegrationPostgres(t *testing.T) {
}
for _, e := range events {
_, err := db.Exec("INSERT INTO event (time_sec, description, tags) VALUES ($1, $2, $3)", e.TimeSec, e.Description, e.Tags)
_, err := p.Exec(t.Context(), "INSERT INTO event (time_sec, description, tags) VALUES ($1, $2, $3)", e.TimeSec, e.Description, e.Tags)
require.NoError(t, err)
}
@@ -1058,7 +1073,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["Deploys"]
@@ -1085,7 +1100,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["Tickets"]
@@ -1108,7 +1123,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1133,7 +1148,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1158,7 +1173,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1184,7 +1199,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1210,7 +1225,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1236,7 +1251,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1262,7 +1277,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1276,8 +1291,20 @@ func TestIntegrationPostgres(t *testing.T) {
})
t.Run("When row limit set to 1", func(t *testing.T) {
dsInfo := sqleng.DataSourceInfo{}
_, handler, err := newPostgres(context.Background(), "error", 1, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
jsonData := sqleng.JsonData{
MaxOpenConns: 10,
MaxIdleConns: 2,
ConnMaxLifetime: 14400,
Timescaledb: false,
Mode: "disable",
ConfigurationMethod: "file-path",
}
dsInfo := sqleng.DataSourceInfo{
JsonData: jsonData,
DecryptedSecureJSONData: map[string]string{},
}
_, handler, err := newPostgres(t.Context(), "error", 1, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
require.NoError(t, err)
@@ -1298,7 +1325,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := handler.QueryData(context.Background(), query)
resp, err := handler.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1328,7 +1355,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := handler.QueryData(context.Background(), query)
resp, err := handler.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
@@ -1344,9 +1371,9 @@ func TestIntegrationPostgres(t *testing.T) {
})
t.Run("Given an empty table", func(t *testing.T) {
_, err := db.Exec("DROP TABLE IF EXISTS empty_obj")
_, err := p.Exec(t.Context(), "DROP TABLE IF EXISTS empty_obj")
require.NoError(t, err)
_, err = db.Exec("CREATE TABLE empty_obj (empty_key VARCHAR(255) NULL, empty_val BIGINT NULL)")
_, err = p.Exec(t.Context(), "CREATE TABLE empty_obj (empty_key VARCHAR(255) NULL, empty_val BIGINT NULL)")
require.NoError(t, err)
t.Run("When no rows are returned, should return an empty frame", func(t *testing.T) {
@@ -1366,7 +1393,7 @@ func TestIntegrationPostgres(t *testing.T) {
},
}
resp, err := exe.QueryData(context.Background(), query)
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
@@ -1376,6 +1403,154 @@ func TestIntegrationPostgres(t *testing.T) {
require.NotNil(t, frames[0].Fields)
require.Empty(t, frames[0].Fields)
})
t.Run("Should handle multiple result sets without panicking", func(t *testing.T) {
// Create a test table for the panic scenario test
sql := `
DROP TABLE IF EXISTS test_multi_results;
CREATE TABLE test_multi_results(
id integer,
name text,
value numeric
);
INSERT INTO test_multi_results VALUES
(1, 'test1', 10.5),
(2, 'test2', 20.7),
(3, 'test3', 30.2);
`
_, err := p.Exec(t.Context(), sql)
require.NoError(t, err)
t.Run("Should handle compatible multiple result sets", func(t *testing.T) {
// This query returns multiple result sets with the same structure
query := &backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
RefID: "A",
JSON: []byte(`{
"rawSql": "SELECT id, name FROM test_multi_results WHERE id <= 2; SELECT id, name FROM test_multi_results WHERE id >= 2;",
"format": "table"
}`),
TimeRange: backend.TimeRange{
From: fromStart,
To: fromStart.Add(1 * time.Hour),
},
},
},
}
// This should not panic and should work correctly
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
frames := queryResult.Frames
require.Len(t, frames, 1)
// The frame should be properly constructed from both SELECT results
frame := frames[0]
require.Equal(t, 2, len(frame.Fields)) // id, name from both queries
require.Equal(t, "id", frame.Fields[0].Name)
require.Equal(t, "name", frame.Fields[1].Name)
require.Equal(t, 4, frame.Rows()) // 2 rows from first result + 2 rows from second result
})
t.Run("Should return error for incompatible multiple result sets", func(t *testing.T) {
// This query returns multiple result sets with different structures - the kind that used to cause panic
query := &backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
RefID: "A",
JSON: []byte(`{
"rawSql": "SELECT id, name FROM test_multi_results WHERE id <= 2; SELECT id, value FROM test_multi_results WHERE id >= 2;",
"format": "table"
}`),
TimeRange: backend.TimeRange{
From: fromStart,
To: fromStart.Add(1 * time.Hour),
},
},
},
}
// This should not panic anymore, but should return an error instead
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
// We expect an error about column mismatch, not a panic
require.Error(t, queryResult.Error)
require.Contains(t, queryResult.Error.Error(), "column name mismatch")
})
t.Run("Should return error for incompatible number of columns", func(t *testing.T) {
// This query returns multiple result sets with different number of columns
// This should fix the error "runtime error: index out of range [1] with length 1"
query := &backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
RefID: "A",
JSON: []byte(`{
"rawSql": "SELECT id, name FROM test_multi_results WHERE id = 1; SELECT id FROM test_multi_results WHERE id = 1;",
"format": "table"
}`),
TimeRange: backend.TimeRange{
From: fromStart,
To: fromStart.Add(1 * time.Hour),
},
},
},
}
// This should not panic anymore, but should return an error instead
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
// We expect an error about incompatible result structure, not a panic
require.Error(t, queryResult.Error)
require.Contains(t, queryResult.Error.Error(), "incompatible result structure: expected 2 columns, got 1 columns")
})
})
t.Run("Should handle queries with mixed statement types", func(t *testing.T) {
// This tests a scenario with UPDATE + SELECT that could cause the original panic
query := &backend.QueryDataRequest{
Queries: []backend.DataQuery{
{
RefID: "A",
JSON: []byte(`{
"rawSql": "UPDATE test_multi_results SET name = 'updated' WHERE id = 1; SELECT id, name FROM test_multi_results WHERE id = 1;",
"format": "table"
}`),
TimeRange: backend.TimeRange{
From: fromStart,
To: fromStart.Add(1 * time.Hour),
},
},
},
}
// This should not panic
resp, err := exe.QueryData(t.Context(), query)
require.NoError(t, err)
queryResult := resp.Responses["A"]
require.NoError(t, queryResult.Error)
frames := queryResult.Frames
require.Len(t, frames, 1)
// Should only contain data from the SELECT part
frame := frames[0]
require.Equal(t, 2, len(frame.Fields)) // id, name
require.Equal(t, 1, frame.Rows()) // 1 row
// Verify the update worked
nameField := frame.Fields[1]
nameValue := nameField.At(0).(*string)
require.Equal(t, "updated", *nameValue)
})
})
t.Run("Test Postgres connection with pgpass file", func(t *testing.T) {
@@ -1385,10 +1560,10 @@ func TestIntegrationPostgres(t *testing.T) {
cnnstr := postgresTestDBConnString()
require.NotContains(t, cnnstr, "password=", "Make sure that password is not in the connection string")
dbPgpass, _, err := newPostgres(context.Background(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
pgpassPool, _, err := newPostgres(t.Context(), "error", 10000, dsInfo, cnnstr, logger, backend.DataSourceInstanceSettings{})
require.NoError(t, err)
_, err = dbPgpass.Exec("SELECT 1") // Test connection
_, err = pgpassPool.Query(t.Context(), "SELECT 1") // Test connection
require.NoError(t, err)
})
}
@@ -3,40 +3,13 @@ package postgres
import (
"context"
"net"
"time"
"github.com/lib/pq"
"golang.org/x/net/proxy"
)
// we wrap the proxy.Dialer to become dialer that the postgres module accepts
func newPostgresProxyDialer(dialer proxy.Dialer) pq.Dialer {
return &postgresProxyDialer{d: dialer}
}
type DialFunc = func(ctx context.Context, network string, address string) (net.Conn, error)
var _ pq.Dialer = (&postgresProxyDialer{})
// postgresProxyDialer implements the postgres dialer using a proxy dialer, as their functions differ slightly
type postgresProxyDialer struct {
d proxy.Dialer
}
// Dial uses the normal proxy dial function with the updated dialer
func (p *postgresProxyDialer) Dial(network, addr string) (c net.Conn, err error) {
return p.d.Dial(network, addr)
}
// DialTimeout uses the normal postgres dial timeout function with the updated dialer
func (p *postgresProxyDialer) DialTimeout(network, address string, timeout time.Duration) (net.Conn, error) {
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
return p.d.(proxy.ContextDialer).DialContext(ctx, network, address)
}
type PgxDialFunc = func(ctx context.Context, network string, address string) (net.Conn, error)
func newPgxDialFunc(dialer proxy.Dialer) PgxDialFunc {
func newDialFunc(dialer proxy.Dialer) DialFunc {
return func(ctx context.Context, network string, addr string) (net.Conn, error) {
return dialer.Dial(network, addr)
}
@@ -1,12 +1,11 @@
package postgres
import (
"database/sql"
"fmt"
"net"
"testing"
"github.com/lib/pq"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/stretchr/testify/require"
"golang.org/x/net/proxy"
)
@@ -25,15 +24,17 @@ func TestPostgresProxyDriver(t *testing.T) {
cnnstr := fmt.Sprintf("postgres://auser:password@%s/db?sslmode=disable", dbURL)
t.Run("Connector should use dialer context that routes through the socks proxy to db", func(t *testing.T) {
connector, err := pq.NewConnector(cnnstr)
config, err := pgxpool.ParseConfig(cnnstr)
require.NoError(t, err)
dialer := newPostgresProxyDialer(&testDialer{})
connector.Dialer(dialer)
config.ConnConfig.DialFunc = newDialFunc(&testDialer{})
db := sql.OpenDB(connector)
err = db.Ping()
pool, err := pgxpool.NewWithConfig(t.Context(), config)
require.NoError(t, err)
defer pool.Close()
err = pool.Ping(t.Context())
require.Error(t, err)
require.Contains(t, err.Error(), "test-dialer is not functional")
})
}
@@ -10,11 +10,10 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/lib/pq"
)
func (e *DataSourceHandler) CheckHealth(ctx context.Context, req *backend.CheckHealthRequest) (*backend.CheckHealthResult, error) {
err := e.Ping()
err := e.Ping(ctx)
if err != nil {
logCheckHealthError(ctx, e.dsInfo, err)
if strings.EqualFold(req.PluginContext.User.Role, "Admin") {
@@ -64,24 +63,6 @@ func ErrToHealthCheckResult(err error) (*backend.CheckHealthResult, error) {
}
}
if errors.Is(err, pq.ErrSSLNotSupported) {
res.Message = "SSL error: Failed to connect to the server"
}
if strings.HasPrefix(err.Error(), "pq") {
res.Message = "Database error: Failed to connect to the postgres server"
if unwrappedErr := errors.Unwrap(err); unwrappedErr != nil {
details["verboseMessage"] = unwrappedErr.Error()
}
}
var pqErr *pq.Error
if errors.As(err, &pqErr) {
if pqErr != nil {
if pqErr.Code != "" {
res.Message += fmt.Sprintf(". Postgres error code: %s", pqErr.Code.Name())
}
details["verboseMessage"] = pqErr.Message
}
}
if errors.Is(err, ErrParsingPostgresURL) {
res.Message = fmt.Sprintf("Connection string error: %s", ErrParsingPostgresURL.Error())
if unwrappedErr := errors.Unwrap(err); unwrappedErr != nil {
@@ -7,7 +7,6 @@ import (
"testing"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/lib/pq"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
@@ -31,15 +30,6 @@ func TestErrToHealthCheckResult(t *testing.T) {
JSONDetails: []byte(`{"errorDetailsLink":"https://grafana.com/docs/grafana/latest/datasources/postgres","verboseMessage":"foo\nread tcp: some op"}`),
},
},
{
name: "db error",
err: errors.Join(errors.New("foo"), &pq.Error{Message: pq.ErrCouldNotDetectUsername.Error(), Code: pq.ErrorCode("28P01")}),
want: &backend.CheckHealthResult{
Status: backend.HealthStatusError,
Message: "foo\npq: pq: Could not detect default username. Please provide one explicitly. Postgres error code: invalid_password",
JSONDetails: []byte(`{"errorDetailsLink":"https://grafana.com/docs/grafana/latest/datasources/postgres","verboseMessage":"pq: Could not detect default username. Please provide one explicitly"}`),
},
},
{
name: "regular error",
err: errors.New("internal server error"),
@@ -2,7 +2,6 @@ package sqleng
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
@@ -19,6 +18,9 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgtype"
"github.com/jackc/pgx/v5/pgxpool"
)
// MetaKeyExecutedQueryString is the key where the executed query should get stored
@@ -81,13 +83,13 @@ type DataPluginConfiguration struct {
type DataSourceHandler struct {
macroEngine SQLMacroEngine
queryResultTransformer SqlQueryResultTransformer
db *sql.DB
timeColumnNames []string
metricColumnTypes []string
log log.Logger
dsInfo DataSourceInfo
rowLimit int64
userError string
pool *pgxpool.Pool
}
type QueryJson struct {
@@ -112,7 +114,7 @@ func (e *DataSourceHandler) TransformQueryError(logger log.Logger, err error) er
return e.queryResultTransformer.TransformQueryError(logger, err)
}
func NewQueryDataHandler(userFacingDefaultError string, db *sql.DB, config DataPluginConfiguration, queryResultTransformer SqlQueryResultTransformer,
func NewQueryDataHandler(userFacingDefaultError string, p *pgxpool.Pool, config DataPluginConfiguration, queryResultTransformer SqlQueryResultTransformer,
macroEngine SQLMacroEngine, log log.Logger) (*DataSourceHandler, error) {
queryDataHandler := DataSourceHandler{
queryResultTransformer: queryResultTransformer,
@@ -132,7 +134,7 @@ func NewQueryDataHandler(userFacingDefaultError string, db *sql.DB, config DataP
queryDataHandler.metricColumnTypes = config.MetricColumnTypes
}
queryDataHandler.db = db
queryDataHandler.pool = p
return &queryDataHandler, nil
}
@@ -143,16 +145,16 @@ type DBDataResponse struct {
func (e *DataSourceHandler) Dispose() {
e.log.Debug("Disposing DB...")
if e.db != nil {
if err := e.db.Close(); err != nil {
e.log.Error("Failed to dispose db", "error", err)
}
if e.pool != nil {
e.pool.Close()
}
e.log.Debug("DB disposed")
}
func (e *DataSourceHandler) Ping() error {
return e.db.Ping()
func (e *DataSourceHandler) Ping(ctx context.Context) error {
return e.pool.Ping(ctx)
}
func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDataRequest) (*backend.QueryDataResponse, error) {
@@ -167,13 +169,13 @@ func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDat
}
err := json.Unmarshal(query.JSON, &queryjson)
if err != nil {
return nil, fmt.Errorf("error unmarshal query json: %w", err)
return nil, backend.DownstreamErrorf("error unmarshal query json: %s", err.Error())
}
// the fill-params are only stored inside this function, during query-interpolation. we do not support
// sending them in "from the outside"
if queryjson.Fill || queryjson.FillInterval != 0.0 || queryjson.FillMode != "" || queryjson.FillValue != 0.0 {
return nil, fmt.Errorf("query fill-parameters not supported")
return nil, backend.DownstreamErrorf("query fill-parameters not supported")
}
if queryjson.RawSql == "" {
@@ -181,7 +183,7 @@ func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDat
}
wg.Add(1)
go e.executeQuery(query, &wg, ctx, ch, queryjson)
go e.executeQuery(ctx, query, &wg, ch, queryjson)
}
wg.Wait()
@@ -196,8 +198,21 @@ func (e *DataSourceHandler) QueryData(ctx context.Context, req *backend.QueryDat
return result, nil
}
func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitGroup, queryContext context.Context,
ch chan DBDataResponse, queryJson QueryJson) {
func (e *DataSourceHandler) execQuery(ctx context.Context, query string) ([]*pgconn.Result, error) {
c, err := e.pool.Acquire(ctx)
if err != nil {
return nil, backend.DownstreamErrorf("failed to acquire connection: %w", err)
}
defer c.Release()
mrr := c.Conn().PgConn().Exec(ctx, query)
// Close returns the first error that occurred during the MultiResultReader's use. We will log that later.
defer mrr.Close() //nolint:errcheck
return mrr.ReadAll()
}
func (e *DataSourceHandler) executeQuery(queryContext context.Context, query backend.DataQuery, wg *sync.WaitGroup,
ch chan DBDataResponse, queryJSON QueryJson) {
defer wg.Done()
queryResult := DBDataResponse{
dataResponse: backend.DataResponse{},
@@ -205,84 +220,77 @@ func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitG
}
logger := e.log.FromContext(queryContext)
defer e.handlePanic(logger, &queryResult, ch)
defer func() {
if r := recover(); r != nil {
logger.Error("ExecuteQuery panic", "error", r, "stack", string(debug.Stack()))
if theErr, ok := r.(error); ok {
queryResult.dataResponse.Error = theErr
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else if theErrString, ok := r.(string); ok {
queryResult.dataResponse.Error = errors.New(theErrString)
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else {
queryResult.dataResponse.Error = fmt.Errorf("unexpected error - %s", e.userError)
queryResult.dataResponse.ErrorSource = backend.ErrorSourceDownstream
}
ch <- queryResult
}
}()
if queryJson.RawSql == "" {
if queryJSON.RawSql == "" {
panic("Query model property rawSql should not be empty at this point")
}
timeRange := query.TimeRange
errAppendDebug := func(frameErr string, err error, query string, source backend.ErrorSource) {
var emptyFrame data.Frame
emptyFrame.SetMeta(&data.FrameMeta{
ExecutedQueryString: query,
})
if isDownstreamError(err) {
source = backend.ErrorSourceDownstream
}
queryResult.dataResponse.Error = fmt.Errorf("%s: %w", frameErr, err)
queryResult.dataResponse.ErrorSource = source
queryResult.dataResponse.Frames = data.Frames{&emptyFrame}
ch <- queryResult
}
// global substitutions
interpolatedQuery := Interpolate(query, timeRange, e.dsInfo.JsonData.TimeInterval, queryJson.RawSql)
interpolatedQuery := Interpolate(query, query.TimeRange, e.dsInfo.JsonData.TimeInterval, queryJSON.RawSql)
// data source specific substitutions
interpolatedQuery, err := e.macroEngine.Interpolate(&query, timeRange, interpolatedQuery)
interpolatedQuery, err := e.macroEngine.Interpolate(&query, query.TimeRange, interpolatedQuery)
if err != nil {
errAppendDebug("interpolation failed", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourcePlugin)
e.handleQueryError("interpolation failed", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
rows, err := e.db.QueryContext(queryContext, interpolatedQuery)
results, err := e.execQuery(queryContext, interpolatedQuery)
if err != nil {
errAppendDebug("db query error", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourceDownstream)
e.handleQueryError("db query error", e.TransformQueryError(logger, err), interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
defer func() {
if err := rows.Close(); err != nil {
logger.Warn("Failed to close rows", "err", err)
qm, err := e.newProcessCfg(queryContext, query, results, interpolatedQuery)
if err != nil {
e.handleQueryError("failed to get configurations", err, interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
frame, err := convertResultsToFrame(results, e.rowLimit)
if err != nil {
e.handleQueryError("convert frame from rows error", err, interpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
e.processFrame(frame, qm, queryResult, ch, logger)
}
func (e *DataSourceHandler) handleQueryError(frameErr string, err error, query string, source backend.ErrorSource, ch chan DBDataResponse, queryResult DBDataResponse) {
var emptyFrame data.Frame
emptyFrame.SetMeta(&data.FrameMeta{ExecutedQueryString: query})
if isDownstreamError(err) {
source = backend.ErrorSourceDownstream
}
queryResult.dataResponse.Error = fmt.Errorf("%s: %w", frameErr, err)
queryResult.dataResponse.ErrorSource = source
queryResult.dataResponse.Frames = data.Frames{&emptyFrame}
ch <- queryResult
}
func (e *DataSourceHandler) handlePanic(logger log.Logger, queryResult *DBDataResponse, ch chan DBDataResponse) {
if r := recover(); r != nil {
logger.Error("ExecuteQuery panic", "error", r, "stack", string(debug.Stack()))
if theErr, ok := r.(error); ok {
queryResult.dataResponse.Error = theErr
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else if theErrString, ok := r.(string); ok {
queryResult.dataResponse.Error = errors.New(theErrString)
queryResult.dataResponse.ErrorSource = backend.ErrorSourcePlugin
} else {
queryResult.dataResponse.Error = fmt.Errorf("unexpected error - %s", e.userError)
queryResult.dataResponse.ErrorSource = backend.ErrorSourceDownstream
}
}()
qm, err := e.newProcessCfg(query, queryContext, rows, interpolatedQuery)
if err != nil {
errAppendDebug("failed to get configurations", err, interpolatedQuery, backend.ErrorSourcePlugin)
return
}
// Convert row.Rows to dataframe
stringConverters := e.queryResultTransformer.GetConverterList()
frame, err := sqlutil.FrameFromRows(rows, e.rowLimit, sqlutil.ToConverters(stringConverters...)...)
if err != nil {
errAppendDebug("convert frame from rows error", err, interpolatedQuery, backend.ErrorSourcePlugin)
return
ch <- *queryResult
}
}
func (e *DataSourceHandler) processFrame(frame *data.Frame, qm *dataQueryModel, queryResult DBDataResponse, ch chan DBDataResponse, logger log.Logger) {
if frame.Meta == nil {
frame.Meta = &data.FrameMeta{}
}
frame.Meta.ExecutedQueryString = interpolatedQuery
frame.Meta.ExecutedQueryString = qm.InterpolatedQuery
// If no rows were returned, clear any previously set `Fields` with a single empty `data.Field` slice.
// Then assign `queryResult.dataResponse.Frames` the current single frame with that single empty Field.
@@ -296,14 +304,14 @@ func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitG
}
if err := convertSQLTimeColumnsToEpochMS(frame, qm); err != nil {
errAppendDebug("converting time columns failed", err, interpolatedQuery, backend.ErrorSourcePlugin)
e.handleQueryError("converting time columns failed", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
if qm.Format == dataQueryFormatSeries {
// time series has to have time column
if qm.timeIndex == -1 {
errAppendDebug("db has no time column", errors.New("time column is missing; make sure your data includes a time column for time series format or switch to a table format that doesn't require it"), interpolatedQuery, backend.ErrorSourceDownstream)
e.handleQueryError("db has no time column", errors.New("time column is missing; make sure your data includes a time column for time series format or switch to a table format that doesn't require it"), qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
@@ -321,7 +329,7 @@ func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitG
var err error
if frame, err = convertSQLValueColumnToFloat(frame, i); err != nil {
errAppendDebug("convert value to float failed", err, interpolatedQuery, backend.ErrorSourcePlugin)
e.handleQueryError("convert value to float failed", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
}
@@ -332,7 +340,7 @@ func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitG
originalData := frame
frame, err = data.LongToWide(frame, qm.FillMissing)
if err != nil {
errAppendDebug("failed to convert long to wide series when converting from dataframe", err, interpolatedQuery, backend.ErrorSourcePlugin)
e.handleQueryError("failed to convert long to wide series when converting from dataframe", err, qm.InterpolatedQuery, backend.ErrorSourceDownstream, ch, queryResult)
return
}
@@ -364,6 +372,7 @@ func (e *DataSourceHandler) executeQuery(query backend.DataQuery, wg *sync.WaitG
if err != nil {
logger.Error("Failed to resample dataframe", "err", err)
frame.AppendNotices(data.Notice{Text: "Failed to resample dataframe", Severity: data.NoticeSeverityWarning})
return
}
}
}
@@ -384,15 +393,38 @@ var Interpolate = func(query backend.DataQuery, timeRange backend.TimeRange, tim
return sql
}
func (e *DataSourceHandler) newProcessCfg(query backend.DataQuery, queryContext context.Context,
rows *sql.Rows, interpolatedQuery string) (*dataQueryModel, error) {
columnNames, err := rows.Columns()
if err != nil {
return nil, err
func (e *DataSourceHandler) newProcessCfg(queryContext context.Context, query backend.DataQuery,
results []*pgconn.Result, interpolatedQuery string) (*dataQueryModel, error) {
// Calculate total number of fields to preallocate slices
totalFields := 0
for _, result := range results {
totalFields += len(result.FieldDescriptions)
}
columnTypes, err := rows.ColumnTypes()
if err != nil {
return nil, err
columnNames := make([]string, 0, totalFields)
columnTypes := make([]string, 0, totalFields)
// The results will contain column information in the metadata
for _, result := range results {
// Get column names from the result metadata
for _, field := range result.FieldDescriptions {
columnNames = append(columnNames, field.Name)
pqtype, ok := pgtype.NewMap().TypeForOID(field.DataTypeOID)
if !ok {
// Handle special cases for field types
switch field.DataTypeOID {
case pgtype.TimetzOID:
columnTypes = append(columnTypes, "timetz")
// money type is 790
case 790:
columnTypes = append(columnTypes, "money")
default:
columnTypes = append(columnTypes, "unknown")
}
} else {
columnTypes = append(columnTypes, pqtype.Name)
}
}
}
qm := &dataQueryModel{
@@ -405,23 +437,23 @@ func (e *DataSourceHandler) newProcessCfg(query backend.DataQuery, queryContext
queryContext: queryContext,
}
queryJson := QueryJson{}
err = json.Unmarshal(query.JSON, &queryJson)
queryJSON := QueryJson{}
err := json.Unmarshal(query.JSON, &queryJSON)
if err != nil {
return nil, err
}
if queryJson.Fill {
if queryJSON.Fill {
qm.FillMissing = &data.FillMissing{}
qm.Interval = time.Duration(queryJson.FillInterval * float64(time.Second))
switch strings.ToLower(queryJson.FillMode) {
qm.Interval = time.Duration(queryJSON.FillInterval * float64(time.Second))
switch strings.ToLower(queryJSON.FillMode) {
case "null":
qm.FillMissing.Mode = data.FillModeNull
case "previous":
qm.FillMissing.Mode = data.FillModePrevious
case "value":
qm.FillMissing.Mode = data.FillModeValue
qm.FillMissing.Value = queryJson.FillValue
qm.FillMissing.Value = queryJSON.FillValue
default:
}
}
@@ -429,13 +461,14 @@ func (e *DataSourceHandler) newProcessCfg(query backend.DataQuery, queryContext
qm.TimeRange.From = query.TimeRange.From.UTC()
qm.TimeRange.To = query.TimeRange.To.UTC()
switch queryJson.Format {
case "time_series":
qm.Format = dataQueryFormatSeries
// Default to time_series if no format is provided
switch queryJSON.Format {
case "table":
qm.Format = dataQueryFormatTable
case "time_series":
fallthrough
default:
panic(fmt.Sprintf("Unrecognized query model format: %q", queryJson.Format))
qm.Format = dataQueryFormatSeries
}
for i, col := range qm.columnNames {
@@ -456,7 +489,7 @@ func (e *DataSourceHandler) newProcessCfg(query backend.DataQuery, queryContext
qm.metricIndex = i
default:
if qm.metricIndex == -1 {
columnType := qm.columnTypes[i].DatabaseTypeName()
columnType := qm.columnTypes[i]
for _, mct := range e.metricColumnTypes {
if columnType == mct {
qm.metricIndex = i
@@ -487,7 +520,7 @@ type dataQueryModel struct {
FillMissing *data.FillMissing // property not set until after Interpolate()
Interval time.Duration
columnNames []string
columnTypes []*sql.ColumnType
columnTypes []string
timeIndex int
timeEndIndex int
metricIndex int
@@ -511,6 +544,219 @@ func convertSQLTimeColumnsToEpochMS(frame *data.Frame, qm *dataQueryModel) error
return nil
}
func convertResultsToFrame(results []*pgconn.Result, rowLimit int64) (*data.Frame, error) {
m := pgtype.NewMap()
// Find the first SELECT result to establish the frame structure
var firstSelectResult *pgconn.Result
for _, result := range results {
if result.CommandTag.Select() {
firstSelectResult = result
break
}
}
// If no SELECT results found, return empty frame
if firstSelectResult == nil {
return data.NewFrame(""), nil
}
// Create frame structure based on the first SELECT result
fields := make(data.Fields, len(firstSelectResult.FieldDescriptions))
fieldTypes, err := getFieldTypesFromDescriptions(firstSelectResult.FieldDescriptions, m)
if err != nil {
return nil, err
}
for i, v := range firstSelectResult.FieldDescriptions {
fields[i] = data.NewFieldFromFieldType(fieldTypes[i], 0)
fields[i].Name = v.Name
}
frame := *data.NewFrame("", fields...)
// Process all SELECT results, but validate column compatibility
for _, result := range results {
// Skip non-select statements
if !result.CommandTag.Select() {
continue
}
// Validate that this result has the same structure as the frame
if len(result.FieldDescriptions) != len(frame.Fields) {
return nil, fmt.Errorf("incompatible result structure: expected %d columns, got %d columns",
len(frame.Fields), len(result.FieldDescriptions))
}
// Validate column names and types match
for i, fd := range result.FieldDescriptions {
if fd.Name != frame.Fields[i].Name {
return nil, fmt.Errorf("column name mismatch at position %d: expected %q, got %q",
i, frame.Fields[i].Name, fd.Name)
}
}
fieldDescriptions := result.FieldDescriptions
for rowIdx := range result.Rows {
if rowIdx == int(rowLimit) {
frame.AppendNotices(data.Notice{
Severity: data.NoticeSeverityWarning,
Text: fmt.Sprintf("Results have been limited to %v because the SQL row limit was reached", rowLimit),
})
break
}
row := make([]any, len(fieldDescriptions))
for colIdx, fd := range fieldDescriptions {
rawValue := result.Rows[rowIdx][colIdx]
if rawValue == nil {
row[colIdx] = nil
continue
}
convertedValue, err := convertPostgresValue(rawValue, fd, m)
if err != nil {
return nil, err
}
row[colIdx] = convertedValue
}
// Validate row length matches frame field count before appending
if len(row) != len(frame.Fields) {
return nil, fmt.Errorf("row data length mismatch: expected %d values, got %d values",
len(frame.Fields), len(row))
}
frame.AppendRow(row...)
}
}
return &frame, nil
}
// convertPostgresValue converts a raw PostgreSQL value to the appropriate Go type
func convertPostgresValue(rawValue []byte, fd pgconn.FieldDescription, m *pgtype.Map) (interface{}, error) {
dataTypeOID := fd.DataTypeOID
format := fd.Format
// Convert based on type
switch fd.DataTypeOID {
case pgtype.Int2OID:
var d *int16
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.Int4OID:
var d *int32
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.Int8OID:
var d *int64
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.NumericOID, pgtype.Float8OID, pgtype.Float4OID:
var d *float64
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.BoolOID:
var d *bool
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.ByteaOID:
d, err := pgtype.ByteaCodec.DecodeValue(pgtype.ByteaCodec{}, m, dataTypeOID, format, rawValue)
if err != nil {
return nil, err
}
str := string(d.([]byte))
return &str, nil
case pgtype.TimestampOID, pgtype.TimestamptzOID, pgtype.DateOID:
var d *time.Time
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.TimeOID, pgtype.TimetzOID:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
case pgtype.JSONOID, pgtype.JSONBOID:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
// Handle null JSON values
if d == nil {
return nil, nil
}
j := json.RawMessage(*d)
return &j, nil
default:
var d *string
scanPlan := m.PlanScan(dataTypeOID, format, &d)
err := scanPlan.Scan(rawValue, &d)
if err != nil {
return nil, err
}
return d, nil
}
}
func getFieldTypesFromDescriptions(fieldDescriptions []pgconn.FieldDescription, m *pgtype.Map) ([]data.FieldType, error) {
fieldTypes := make([]data.FieldType, len(fieldDescriptions))
for i, v := range fieldDescriptions {
typeName, ok := m.TypeForOID(v.DataTypeOID)
if !ok {
fieldTypes[i] = data.FieldTypeNullableString
} else {
switch typeName.Name {
case "int2":
fieldTypes[i] = data.FieldTypeNullableInt16
case "int4":
fieldTypes[i] = data.FieldTypeNullableInt32
case "int8":
fieldTypes[i] = data.FieldTypeNullableInt64
case "float4", "float8", "numeric":
fieldTypes[i] = data.FieldTypeNullableFloat64
case "bool":
fieldTypes[i] = data.FieldTypeNullableBool
case "timestamptz", "timestamp", "date":
fieldTypes[i] = data.FieldTypeNullableTime
case "json", "jsonb":
fieldTypes[i] = data.FieldTypeNullableJSON
default:
fieldTypes[i] = data.FieldTypeNullableString
}
}
}
return fieldTypes, nil
}
// convertSQLTimeColumnToEpochMS converts column named time to unix timestamp in milliseconds
// to make native datetime types and epoch dates work in annotation and table queries.
func convertSQLTimeColumnToEpochMS(frame *data.Frame, timeIndex int) error {
@@ -9,6 +9,8 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/data"
"github.com/grafana/grafana-plugin-sdk-go/data/sqlutil"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgtype"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
@@ -425,6 +427,246 @@ func TestSQLEngine(t *testing.T) {
})
}
func TestConvertResultsToFrame(t *testing.T) {
// Import the pgx packages needed for testing
// These imports are included in the main file but need to be accessible for tests
t.Run("convertResultsToFrame with single result", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
{Name: "value", DataTypeOID: pgtype.Float8OID},
}
// Create mock result data
mockRows := [][][]byte{
{[]byte("1"), []byte("test1"), []byte("10.5")},
{[]byte("2"), []byte("test2"), []byte("20.7")},
}
// Create mock result
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 3, len(frame.Fields))
require.Equal(t, 2, frame.Rows())
// Verify field names
require.Equal(t, "id", frame.Fields[0].Name)
require.Equal(t, "name", frame.Fields[1].Name)
require.Equal(t, "value", frame.Fields[2].Name)
})
t.Run("convertResultsToFrame with multiple compatible results", func(t *testing.T) {
// Create mock field descriptions (same structure for both results)
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
}
// Create first result
mockRows1 := [][][]byte{
{[]byte("1"), []byte("test1")},
{[]byte("2"), []byte("test2")},
}
result1 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows1,
}
result1.CommandTag = pgconn.NewCommandTag("SELECT 2")
// Create second result with same structure
mockRows2 := [][][]byte{
{[]byte("3"), []byte("test3")},
{[]byte("4"), []byte("test4")},
}
result2 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows2,
}
result2.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result1, result2}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 2, len(frame.Fields))
require.Equal(t, 4, frame.Rows()) // Should have rows from both results
// Verify field names
require.Equal(t, "id", frame.Fields[0].Name)
require.Equal(t, "name", frame.Fields[1].Name)
})
t.Run("convertResultsToFrame with row limit", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
// Create mock result data with 3 rows
mockRows := [][][]byte{
{[]byte("1")},
{[]byte("2")},
{[]byte("3")},
}
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 3")
results := []*pgconn.Result{result}
// Set row limit to 2
frame, err := convertResultsToFrame(results, 2)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 2, frame.Rows()) // Should be limited to 2 rows
// Should have a notice about the limit
require.NotNil(t, frame.Meta)
require.Len(t, frame.Meta.Notices, 1)
require.Contains(t, frame.Meta.Notices[0].Text, "Results have been limited to 2")
})
t.Run("convertResultsToFrame with mixed SELECT and non-SELECT results", func(t *testing.T) {
// Create a non-SELECT result (should be skipped)
nonSelectResult := &pgconn.Result{}
nonSelectResult.CommandTag = pgconn.NewCommandTag("UPDATE 1")
// Create a SELECT result
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
mockRows := [][][]byte{
{[]byte("1")},
}
selectResult := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
selectResult.CommandTag = pgconn.NewCommandTag("SELECT 1")
results := []*pgconn.Result{nonSelectResult, selectResult}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 1, frame.Rows())
})
t.Run("convertResultsToFrame with no SELECT results", func(t *testing.T) {
// Create only non-SELECT results
result1 := &pgconn.Result{}
result1.CommandTag = pgconn.NewCommandTag("UPDATE 1")
result2 := &pgconn.Result{}
result2.CommandTag = pgconn.NewCommandTag("INSERT 1")
results := []*pgconn.Result{result1, result2}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 0, len(frame.Fields))
require.Equal(t, 0, frame.Rows())
})
t.Run("convertResultsToFrame with multiple results and row limit per result", func(t *testing.T) {
// Create mock field descriptions (same structure for both results)
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
}
// Create first result with 3 rows
mockRows1 := [][][]byte{
{[]byte("1")},
{[]byte("2")},
{[]byte("3")},
}
result1 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows1,
}
result1.CommandTag = pgconn.NewCommandTag("SELECT 3")
// Create second result with 3 rows
mockRows2 := [][][]byte{
{[]byte("4")},
{[]byte("5")},
{[]byte("6")},
}
result2 := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows2,
}
result2.CommandTag = pgconn.NewCommandTag("SELECT 3")
results := []*pgconn.Result{result1, result2}
// Set row limit to 2 (should limit each result to 2 rows)
frame, err := convertResultsToFrame(results, 2)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 1, len(frame.Fields))
require.Equal(t, 4, frame.Rows()) // 2 rows from each result
// Should have notices about the limit from both results
require.NotNil(t, frame.Meta)
require.Len(t, frame.Meta.Notices, 2)
require.Contains(t, frame.Meta.Notices[0].Text, "Results have been limited to 2")
require.Contains(t, frame.Meta.Notices[1].Text, "Results have been limited to 2")
})
t.Run("convertResultsToFrame handles null values correctly", func(t *testing.T) {
// Create mock field descriptions
fieldDescs := []pgconn.FieldDescription{
{Name: "id", DataTypeOID: pgtype.Int4OID},
{Name: "name", DataTypeOID: pgtype.TextOID},
}
// Create mock result data with null values
mockRows := [][][]byte{
{[]byte("1"), nil}, // null name
{nil, []byte("test2")}, // null id
}
result := &pgconn.Result{
FieldDescriptions: fieldDescs,
Rows: mockRows,
}
result.CommandTag = pgconn.NewCommandTag("SELECT 2")
results := []*pgconn.Result{result}
frame, err := convertResultsToFrame(results, 1000)
require.NoError(t, err)
require.NotNil(t, frame)
require.Equal(t, 2, len(frame.Fields))
require.Equal(t, 2, frame.Rows())
// Check that null values are handled correctly
// The exact representation depends on the field type, but should not panic
require.NotPanics(t, func() {
frame.Fields[0].At(1) // null id
frame.Fields[1].At(0) // null name
})
})
}
type testQueryResultTransformer struct {
transformQueryErrorWasCalled bool
}
@@ -6,16 +6,13 @@ import (
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/datasource"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/setting"
postgres "github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource"
)
func main() {
// No need to pass logger name, it will be set by the plugin SDK
logger := backend.NewLoggerWith()
// TODO: get rid of setting.NewCfg() once PostgresDSUsePGX is removed
cfg := setting.NewCfg()
if err := datasource.Manage("grafana-postgresql-datasource", postgres.NewInstanceSettings(logger, cfg.DataPath), datasource.ManageOpts{}); err != nil {
if err := datasource.Manage("grafana-postgresql-datasource", postgres.NewInstanceSettings(logger), datasource.ManageOpts{}); err != nil {
log.DefaultLogger.Error(err.Error())
os.Exit(1)
}
@@ -9,14 +9,14 @@
// }
// Name:
// Dimensions: 12 Fields by 2 Rows
// +----------------------------------------+----------------------------------------+--------------------------------------+--------------------------------------+---------------------------------+---------------------------------+--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-----------------+-----------------+
// | Name: ts | Name: tsnn | Name: tsz | Name: tsznn | Name: d | Name: dnn | Name: t | Name: tnn | Name: tz | Name: tznn | Name: i | Name: inn |
// | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: |
// | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*string | Type: []*string |
// +----------------------------------------+----------------------------------------+--------------------------------------+--------------------------------------+---------------------------------+---------------------------------+--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-----------------+-----------------+
// | 2023-11-15 05:06:07.123456 +0000 +0000 | 2023-11-15 05:06:08.123456 +0000 +0000 | 2021-07-22 11:22:33.654321 +0000 UTC | 2021-07-22 11:22:34.654321 +0000 UTC | 2023-12-20 00:00:00 +0000 +0000 | 2023-12-21 00:00:00 +0000 +0000 | 0000-01-01 12:34:56.234567 +0000 UTC | 0000-01-01 12:34:57.234567 +0000 UTC | 0000-01-01 23:12:36.765432 +0100 +0100 | 0000-01-01 23:12:37.765432 +0100 +0100 | 00:00:00.987654 | 00:00:00.887654 |
// | null | 2023-11-15 05:06:09.123456 +0000 +0000 | null | 2021-07-22 11:22:35.654321 +0000 UTC | null | 2023-12-22 00:00:00 +0000 +0000 | null | 0000-01-01 12:34:58.234567 +0000 UTC | null | 0000-01-01 23:12:38.765432 +0100 +0100 | null | 00:00:00.787654 |
// +----------------------------------------+----------------------------------------+--------------------------------------+--------------------------------------+---------------------------------+---------------------------------+--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-----------------+-----------------+
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
// | Name: ts | Name: tsnn | Name: tsz | Name: tsznn | Name: d | Name: dnn | Name: t | Name: tnn | Name: tz | Name: tznn | Name: i | Name: inn |
// | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: |
// | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*string | Type: []*string | Type: []*string | Type: []*string | Type: []*string | Type: []*string |
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
// | 2023-11-15 05:06:07.123456 +0000 UTC | 2023-11-15 05:06:08.123456 +0000 UTC | 2021-07-22 11:22:33.654321 +0000 +0000 | 2021-07-22 11:22:34.654321 +0000 +0000 | 2023-12-20 00:00:00 +0000 UTC | 2023-12-21 00:00:00 +0000 UTC | 12:34:56.234567 | 12:34:57.234567 | 23:12:36.765432+01 | 23:12:37.765432+01 | 00:00:00.987654 | 00:00:00.887654 |
// | null | 2023-11-15 05:06:09.123456 +0000 UTC | null | 2021-07-22 11:22:35.654321 +0000 +0000 | null | 2023-12-22 00:00:00 +0000 UTC | null | 12:34:58.234567 | null | 23:12:38.765432+01 | null | 00:00:00.787654 |
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
//
//
// 🌟 This was machine generated. Do not edit. 🌟
@@ -83,33 +83,33 @@
},
{
"name": "t",
"type": "time",
"type": "string",
"typeInfo": {
"frame": "time.Time",
"frame": "string",
"nullable": true
}
},
{
"name": "tnn",
"type": "time",
"type": "string",
"typeInfo": {
"frame": "time.Time",
"frame": "string",
"nullable": true
}
},
{
"name": "tz",
"type": "time",
"type": "string",
"typeInfo": {
"frame": "time.Time",
"frame": "string",
"nullable": true
}
},
{
"name": "tznn",
"type": "time",
"type": "string",
"typeInfo": {
"frame": "time.Time",
"frame": "string",
"nullable": true
}
},
@@ -158,20 +158,20 @@
1703203200000
],
[
-62167173903766,
"12:34:56.234567",
null
],
[
-62167173902766,
-62167173901766
"12:34:57.234567",
"12:34:58.234567"
],
[
-62167139243235,
"23:12:36.765432+01",
null
],
[
-62167139242235,
-62167139241235
"23:12:37.765432+01",
"23:12:38.765432+01"
],
[
"00:00:00.987654",
@@ -201,22 +201,10 @@
],
null,
null,
[
567000,
0
],
[
567000,
567000
],
[
432000,
0
],
[
432000,
432000
],
null,
null,
null,
null,
null,
null
]
@@ -1,214 +0,0 @@
// 🌟 This was machine generated. Do not edit. 🌟
//
// Frame[0] {
// "typeVersion": [
// 0,
// 0
// ],
// "executedQueryString": "SELECT * FROM tbl"
// }
// Name:
// Dimensions: 12 Fields by 2 Rows
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
// | Name: ts | Name: tsnn | Name: tsz | Name: tsznn | Name: d | Name: dnn | Name: t | Name: tnn | Name: tz | Name: tznn | Name: i | Name: inn |
// | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: | Labels: |
// | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*time.Time | Type: []*string | Type: []*string | Type: []*string | Type: []*string | Type: []*string | Type: []*string |
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
// | 2023-11-15 05:06:07.123456 +0000 UTC | 2023-11-15 05:06:08.123456 +0000 UTC | 2021-07-22 11:22:33.654321 +0000 +0000 | 2021-07-22 11:22:34.654321 +0000 +0000 | 2023-12-20 00:00:00 +0000 UTC | 2023-12-21 00:00:00 +0000 UTC | 12:34:56.234567 | 12:34:57.234567 | 23:12:36.765432+01 | 23:12:37.765432+01 | 00:00:00.987654 | 00:00:00.887654 |
// | null | 2023-11-15 05:06:09.123456 +0000 UTC | null | 2021-07-22 11:22:35.654321 +0000 +0000 | null | 2023-12-22 00:00:00 +0000 UTC | null | 12:34:58.234567 | null | 23:12:38.765432+01 | null | 00:00:00.787654 |
// +--------------------------------------+--------------------------------------+----------------------------------------+----------------------------------------+-------------------------------+-------------------------------+-----------------+-----------------+--------------------+--------------------+-----------------+-----------------+
//
//
// 🌟 This was machine generated. Do not edit. 🌟
{
"status": 200,
"frames": [
{
"schema": {
"meta": {
"typeVersion": [
0,
0
],
"executedQueryString": "SELECT * FROM tbl"
},
"fields": [
{
"name": "ts",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "tsnn",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "tsz",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "tsznn",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "d",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "dnn",
"type": "time",
"typeInfo": {
"frame": "time.Time",
"nullable": true
}
},
{
"name": "t",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
},
{
"name": "tnn",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
},
{
"name": "tz",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
},
{
"name": "tznn",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
},
{
"name": "i",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
},
{
"name": "inn",
"type": "string",
"typeInfo": {
"frame": "string",
"nullable": true
}
}
]
},
"data": {
"values": [
[
1700024767123,
null
],
[
1700024768123,
1700024769123
],
[
1626952953654,
null
],
[
1626952954654,
1626952955654
],
[
1703030400000,
null
],
[
1703116800000,
1703203200000
],
[
"12:34:56.234567",
null
],
[
"12:34:57.234567",
"12:34:58.234567"
],
[
"23:12:36.765432+01",
null
],
[
"23:12:37.765432+01",
"23:12:38.765432+01"
],
[
"00:00:00.987654",
null
],
[
"00:00:00.887654",
"00:00:00.787654"
]
],
"nanos": [
[
456000,
0
],
[
456000,
456000
],
[
321000,
0
],
[
321000,
321000
],
null,
null,
null,
null,
null,
null,
null,
null
]
}
}
]
}
@@ -1,44 +0,0 @@
-- SELECT * FROM tbl
-- test all date/time-based postgres data types
CREATE TEMPORARY TABLE tbl (
ts timestamp,
tsnn timestamp NOT NULL,
tsz timestamp with time zone,
tsznn timestamp with time zone NOT NULL,
d date,
dnn date NOT NULL,
t time,
tnn time NOT NULL,
tz time with time zone,
tznn time with time zone NOT NULL,
i interval,
inn interval NOT NULL
);
INSERT INTO tbl (ts, tsnn, tsz, tsznn, d, dnn, t, tnn, tz, tznn, i, inn) VALUES (
'2023-11-15 05:06:07.123456',
'2023-11-15 05:06:08.123456',
'2021-07-22 13:22:33.654321 Europe/Berlin',
'2021-07-22 13:22:34.654321 Europe/Berlin',
'2023-12-20',
'2023-12-21',
'12:34:56.234567',
'12:34:57.234567',
'23:12:36.765432+1',
'23:12:37.765432+1',
'987654 microsecond',
'887654 microsecond'
), (
NULL,
'2023-11-15 05:06:09.123456',
NULL,
'2021-07-22 13:22:35.654321 Europe/Berlin',
NULL,
'2023-12-22',
NULL,
'12:34:58.234567',
NULL,
'23:12:38.765432+1',
NULL,
'787654 microsecond'
);
@@ -3,48 +3,15 @@ package postgres
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
)
var validateCertFunc = validateCertFilePaths
var writeCertFileFunc = writeCertFile
type certFileType int
const (
rootCert = iota
clientCert
clientKey
)
type tlsSettingsProvider interface {
getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error)
}
type datasourceCacheManager struct {
locker *locker
cache sync.Map
}
type tlsManager struct {
logger log.Logger
dsCacheInstance datasourceCacheManager
dataPath string
}
func newTLSManager(logger log.Logger, dataPath string) tlsSettingsProvider {
return &tlsManager{
logger: logger,
dataPath: dataPath,
dsCacheInstance: datasourceCacheManager{locker: newLocker()},
}
type TLSManager struct {
logger log.Logger
}
type tlsSettings struct {
@@ -55,178 +22,122 @@ type tlsSettings struct {
CertKeyFile string
}
func (m *tlsManager) getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error) {
tlsconfig := tlsSettings{
func newTLSManager(logger log.Logger) *TLSManager {
return &TLSManager{
logger: logger,
}
}
// getTLSSettings retrieves TLS settings and handles certificate file creation if needed.
func (m *TLSManager) getTLSSettings(dsInfo sqleng.DataSourceInfo) (tlsSettings, error) {
tlsConfig := tlsSettings{
Mode: dsInfo.JsonData.Mode,
}
isTLSDisabled := (tlsconfig.Mode == "disable")
if isTLSDisabled {
if tlsConfig.Mode == "disable" {
m.logger.Debug("Postgres TLS/SSL is disabled")
return tlsconfig, nil
return tlsConfig, nil
}
m.logger.Debug("Postgres TLS/SSL is enabled", "tlsMode", tlsconfig.Mode)
tlsConfig.ConfigurationMethod = dsInfo.JsonData.ConfigurationMethod
tlsConfig.RootCertFile = dsInfo.JsonData.RootCertFile
tlsConfig.CertFile = dsInfo.JsonData.CertFile
tlsConfig.CertKeyFile = dsInfo.JsonData.CertKeyFile
tlsconfig.ConfigurationMethod = dsInfo.JsonData.ConfigurationMethod
tlsconfig.RootCertFile = dsInfo.JsonData.RootCertFile
tlsconfig.CertFile = dsInfo.JsonData.CertFile
tlsconfig.CertKeyFile = dsInfo.JsonData.CertKeyFile
if tlsconfig.ConfigurationMethod == "file-content" {
if err := m.writeCertFiles(dsInfo, &tlsconfig); err != nil {
return tlsconfig, err
if tlsConfig.ConfigurationMethod == "file-content" {
if err := m.createCertFiles(dsInfo, &tlsConfig); err != nil {
return tlsConfig, fmt.Errorf("failed to create TLS certificate files: %w", err)
}
} else {
if err := validateCertFunc(tlsconfig.RootCertFile, tlsconfig.CertFile, tlsconfig.CertKeyFile); err != nil {
return tlsconfig, err
if err := validateCertFunc(tlsConfig.RootCertFile, tlsConfig.CertFile, tlsConfig.CertKeyFile); err != nil {
return tlsConfig, fmt.Errorf("invalid TLS certificate file paths: %w", err)
}
}
return tlsconfig, nil
return tlsConfig, nil
}
func (t certFileType) String() string {
switch t {
case rootCert:
return "root certificate"
case clientCert:
return "client certificate"
case clientKey:
return "client key"
default:
panic(fmt.Sprintf("Unrecognized certFileType %d", t))
}
}
// createCertFiles writes certificate files to temporary locations.
func (m *TLSManager) createCertFiles(dsInfo sqleng.DataSourceInfo, tlsConfig *tlsSettings) error {
m.logger.Debug("Writing TLS certificate files to temporary locations")
func getFileName(dataDir string, fileType certFileType) string {
var filename string
switch fileType {
case rootCert:
filename = "root.crt"
case clientCert:
filename = "client.crt"
case clientKey:
filename = "client.key"
default:
panic(fmt.Sprintf("unrecognized certFileType %s", fileType.String()))
}
generatedFilePath := filepath.Join(dataDir, filename)
return generatedFilePath
}
// writeCertFile writes a certificate file.
func writeCertFile(logger log.Logger, fileContent string, generatedFilePath string) error {
fileContent = strings.TrimSpace(fileContent)
if fileContent != "" {
logger.Debug("Writing cert file", "path", generatedFilePath)
if err := os.WriteFile(generatedFilePath, []byte(fileContent), 0600); err != nil {
return err
}
// Make sure the file has the permissions expected by the Postgresql driver, otherwise it will bail
if err := os.Chmod(generatedFilePath, 0600); err != nil {
return err
}
return nil
}
logger.Debug("Deleting cert file since no content is provided", "path", generatedFilePath)
exists, err := fileExists(generatedFilePath)
if err != nil {
var err error
if tlsConfig.RootCertFile, err = m.writeCertFile("root-*.crt", dsInfo.DecryptedSecureJSONData["tlsCACert"]); err != nil {
return err
}
if exists {
if err := os.Remove(generatedFilePath); err != nil {
return fmt.Errorf("failed to remove %q: %w", generatedFilePath, err)
}
if tlsConfig.CertFile, err = m.writeCertFile("client-*.crt", dsInfo.DecryptedSecureJSONData["tlsClientCert"]); err != nil {
return err
}
if tlsConfig.CertKeyFile, err = m.writeCertFile("client-*.key", dsInfo.DecryptedSecureJSONData["tlsClientKey"]); err != nil {
return err
}
return nil
}
func (m *tlsManager) writeCertFiles(dsInfo sqleng.DataSourceInfo, tlsconfig *tlsSettings) error {
m.logger.Debug("Writing TLS certificate files to disk")
tlsRootCert := dsInfo.DecryptedSecureJSONData["tlsCACert"]
tlsClientCert := dsInfo.DecryptedSecureJSONData["tlsClientCert"]
tlsClientKey := dsInfo.DecryptedSecureJSONData["tlsClientKey"]
if tlsRootCert == "" && tlsClientCert == "" && tlsClientKey == "" {
m.logger.Debug("No TLS/SSL certificates provided")
// writeCertFile writes a single certificate file to a temporary location.
func (m *TLSManager) writeCertFile(pattern, content string) (string, error) {
if content == "" {
return "", nil
}
// Calculate all files path
workDir := filepath.Join(m.dataPath, "tls", dsInfo.UID+"generatedTLSCerts")
tlsconfig.RootCertFile = getFileName(workDir, rootCert)
tlsconfig.CertFile = getFileName(workDir, clientCert)
tlsconfig.CertKeyFile = getFileName(workDir, clientKey)
// Find datasource in the cache, if found, skip writing files
cacheKey := strconv.Itoa(int(dsInfo.ID))
m.dsCacheInstance.locker.RLock(cacheKey)
item, ok := m.dsCacheInstance.cache.Load(cacheKey)
m.dsCacheInstance.locker.RUnlock(cacheKey)
if ok {
if !item.(time.Time).Before(dsInfo.Updated) {
return nil
}
}
m.dsCacheInstance.locker.Lock(cacheKey)
defer m.dsCacheInstance.locker.Unlock(cacheKey)
item, ok = m.dsCacheInstance.cache.Load(cacheKey)
if ok {
if !item.(time.Time).Before(dsInfo.Updated) {
return nil
}
}
// Write certification directory and files
exists, err := fileExists(workDir)
m.logger.Debug("Writing certificate file", "pattern", pattern)
file, err := os.CreateTemp("", pattern)
if err != nil {
return err
return "", fmt.Errorf("failed to create temporary file: %w", err)
}
if !exists {
if err := os.MkdirAll(workDir, 0700); err != nil {
return err
defer func() {
if err := file.Close(); err != nil {
m.logger.Error("Failed to close file", "error", err)
}
}()
if _, err := file.WriteString(content); err != nil {
return "", fmt.Errorf("failed to write to temporary file: %w", err)
}
if err = writeCertFileFunc(m.logger, tlsRootCert, tlsconfig.RootCertFile); err != nil {
return err
}
if err = writeCertFileFunc(m.logger, tlsClientCert, tlsconfig.CertFile); err != nil {
return err
}
if err = writeCertFileFunc(m.logger, tlsClientKey, tlsconfig.CertKeyFile); err != nil {
return err
}
// we do not want to point to cert-files that do not exist
if tlsRootCert == "" {
tlsconfig.RootCertFile = ""
}
if tlsClientCert == "" {
tlsconfig.CertFile = ""
}
if tlsClientKey == "" {
tlsconfig.CertKeyFile = ""
}
// Update datasource cache
m.dsCacheInstance.cache.Store(cacheKey, dsInfo.Updated)
return nil
return file.Name(), nil
}
// validateCertFilePaths validates configured certificate file paths.
func validateCertFilePaths(rootCert, clientCert, clientKey string) error {
for _, fpath := range []string{rootCert, clientCert, clientKey} {
if fpath == "" {
// cleanupCertFiles removes temporary certificate files.
func (m *TLSManager) cleanupCertFiles(tlsConfig tlsSettings) {
// Only clean up if the configuration method is "file-content"
if tlsConfig.ConfigurationMethod != "file-content" {
m.logger.Debug("Skipping cleanup of TLS certificate files")
return
}
m.logger.Debug("Cleaning up TLS certificate files")
files := []struct {
path string
name string
}{
{tlsConfig.RootCertFile, "root certificate"},
{tlsConfig.CertFile, "client certificate"},
{tlsConfig.CertKeyFile, "client key"},
}
for _, file := range files {
if file.path == "" {
continue
}
exists, err := fileExists(fpath)
if err := os.Remove(file.path); err != nil {
m.logger.Error("Failed to remove file", "type", file.name, "path", file.path, "error", err)
} else {
m.logger.Debug("Successfully removed file", "type", file.name, "path", file.path)
}
}
}
// validateCertFilePaths validates the existence of configured certificate file paths.
func validateCertFilePaths(rootCert, clientCert, clientKey string) error {
for _, path := range []string{rootCert, clientCert, clientKey} {
if path == "" {
continue
}
exists, err := fileExists(path)
if err != nil {
return err
return fmt.Errorf("error checking file existence: %w", err)
}
if !exists {
return sqleng.ErrCertFileNotExist
@@ -235,15 +146,14 @@ func validateCertFilePaths(rootCert, clientCert, clientKey string) error {
return nil
}
// Exists determines whether a file/directory exists or not.
func fileExists(fpath string) (bool, error) {
_, err := os.Stat(fpath)
// fileExists checks if a file exists at the given path.
func fileExists(path string) (bool, error) {
_, err := os.Stat(path)
if err != nil {
if !os.IsNotExist(err) {
return false, err
if os.IsNotExist(err) {
return false, nil
}
return false, nil
return false, err
}
return true, nil
}
@@ -2,176 +2,21 @@ package postgres
import (
"fmt"
"os"
"path/filepath"
"strconv"
"strings"
"sync"
"testing"
"time"
"github.com/grafana/grafana-plugin-sdk-go/backend"
"github.com/grafana/grafana-plugin-sdk-go/backend/log"
"github.com/grafana/grafana/pkg/setting"
"github.com/grafana/grafana/pkg/tsdb/grafana-postgresql-datasource/sqleng"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
_ "github.com/lib/pq"
)
var writeCertFileCallNum int
// TestDataSourceCacheManager is to test the Cache manager
func TestDataSourceCacheManager(t *testing.T) {
cfg := setting.NewCfg()
cfg.DataPath = t.TempDir()
mng := tlsManager{
logger: backend.NewLoggerWith("logger", "tsdb.postgres"),
dsCacheInstance: datasourceCacheManager{locker: newLocker()},
dataPath: cfg.DataPath,
}
jsonData := sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-content",
}
secureJSONData := map[string]string{
"tlsClientCert": "I am client certification",
"tlsClientKey": "I am client key",
"tlsCACert": "I am CA certification",
}
updateTime := time.Now().Add(-5 * time.Minute)
mockValidateCertFilePaths()
t.Cleanup(resetValidateCertFilePaths)
t.Run("Check datasource cache creation", func(t *testing.T) {
var wg sync.WaitGroup
wg.Add(10)
for id := int64(1); id <= 10; id++ {
go func(id int64) {
ds := sqleng.DataSourceInfo{
ID: id,
Updated: updateTime,
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(ds, &s)
require.NoError(t, err)
wg.Done()
}(id)
}
wg.Wait()
t.Run("check cache creation is succeed", func(t *testing.T) {
for id := int64(1); id <= 10; id++ {
updated, ok := mng.dsCacheInstance.cache.Load(strconv.Itoa(int(id)))
require.True(t, ok)
require.Equal(t, updateTime, updated)
}
})
})
t.Run("Check datasource cache modification", func(t *testing.T) {
t.Run("check when version not changed, cache and files are not updated", func(t *testing.T) {
mockWriteCertFile()
t.Cleanup(resetWriteCertFile)
var wg1 sync.WaitGroup
wg1.Add(5)
for id := int64(1); id <= 5; id++ {
go func(id int64) {
ds := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime,
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(ds, &s)
require.NoError(t, err)
wg1.Done()
}(id)
}
wg1.Wait()
assert.Equal(t, writeCertFileCallNum, 0)
})
t.Run("cache is updated with the last datasource version", func(t *testing.T) {
dsV2 := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime.Add(time.Minute),
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
dsV3 := sqleng.DataSourceInfo{
ID: 1,
Updated: updateTime.Add(2 * time.Minute),
Database: "database",
JsonData: jsonData,
DecryptedSecureJSONData: secureJSONData,
UID: "testData",
}
s := tlsSettings{}
err := mng.writeCertFiles(dsV2, &s)
require.NoError(t, err)
err = mng.writeCertFiles(dsV3, &s)
require.NoError(t, err)
version, ok := mng.dsCacheInstance.cache.Load("1")
require.True(t, ok)
require.Equal(t, updateTime.Add(2*time.Minute), version)
})
})
}
// Test getFileName
func TestGetFileName(t *testing.T) {
testCases := []struct {
desc string
datadir string
fileType certFileType
expErr string
expectedGeneratedPath string
}{
{
desc: "Get File Name for root certification",
datadir: ".",
fileType: rootCert,
expectedGeneratedPath: "root.crt",
},
{
desc: "Get File Name for client certification",
datadir: ".",
fileType: clientCert,
expectedGeneratedPath: "client.crt",
},
{
desc: "Get File Name for client certification",
datadir: ".",
fileType: clientKey,
expectedGeneratedPath: "client.key",
},
}
for _, tt := range testCases {
t.Run(tt.desc, func(t *testing.T) {
generatedPath := getFileName(tt.datadir, tt.fileType)
assert.Equal(t, tt.expectedGeneratedPath, generatedPath)
})
}
}
// Test getTLSSettings.
func TestGetTLSSettings(t *testing.T) {
cfg := setting.NewCfg()
cfg.DataPath = t.TempDir()
mockValidateCertFilePaths()
t.Cleanup(resetValidateCertFilePaths)
@@ -216,75 +61,13 @@ func TestGetTLSSettings(t *testing.T) {
CertKeyFile: "i/am/coding/client.key",
},
},
{
desc: "Custom TLS mode verify-full with certificate files content",
updated: updatedTime.Add(2 * time.Minute),
uid: "xxx",
jsonData: sqleng.JsonData{
Mode: "verify-full",
ConfigurationMethod: "file-content",
},
secureJSONData: map[string]string{
"tlsCACert": "I am CA certification",
"tlsClientCert": "I am client certification",
"tlsClientKey": "I am client key",
},
tlsSettings: tlsSettings{
Mode: "verify-full",
ConfigurationMethod: "file-content",
RootCertFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "root.crt"),
CertFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "client.crt"),
CertKeyFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "client.key"),
},
},
{
desc: "Custom TLS mode verify-ca with no client certificates with certificate files content",
updated: updatedTime.Add(3 * time.Minute),
uid: "xxx",
jsonData: sqleng.JsonData{
Mode: "verify-ca",
ConfigurationMethod: "file-content",
},
secureJSONData: map[string]string{
"tlsCACert": "I am CA certification",
},
tlsSettings: tlsSettings{
Mode: "verify-ca",
ConfigurationMethod: "file-content",
RootCertFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "root.crt"),
CertFile: "",
CertKeyFile: "",
},
},
{
desc: "Custom TLS mode require with client certificates and no root certificate with certificate files content",
updated: updatedTime.Add(4 * time.Minute),
uid: "xxx",
jsonData: sqleng.JsonData{
Mode: "require",
ConfigurationMethod: "file-content",
},
secureJSONData: map[string]string{
"tlsClientCert": "I am client certification",
"tlsClientKey": "I am client key",
},
tlsSettings: tlsSettings{
Mode: "require",
ConfigurationMethod: "file-content",
RootCertFile: "",
CertFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "client.crt"),
CertKeyFile: filepath.Join(cfg.DataPath, "tls", "xxxgeneratedTLSCerts", "client.key"),
},
},
}
for _, tt := range testCases {
t.Run(tt.desc, func(t *testing.T) {
var settings tlsSettings
var err error
mng := tlsManager{
logger: backend.NewLoggerWith("logger", "tsdb.postgres"),
dsCacheInstance: datasourceCacheManager{locker: newLocker()},
dataPath: cfg.DataPath,
mng := TLSManager{
logger: backend.NewLoggerWith("logger", "tsdb.postgres"),
}
ds := sqleng.DataSourceInfo{
@@ -318,15 +101,145 @@ func resetValidateCertFilePaths() {
validateCertFunc = validateCertFilePaths
}
func mockWriteCertFile() {
writeCertFileCallNum = 0
writeCertFileFunc = func(logger log.Logger, fileContent string, generatedFilePath string) error {
writeCertFileCallNum++
return nil
func TestTLSManager_GetTLSSettings(t *testing.T) {
logger := log.New()
tlsManager := newTLSManager(logger)
dsInfo := sqleng.DataSourceInfo{
JsonData: sqleng.JsonData{
Mode: "require",
ConfigurationMethod: "file-content",
},
DecryptedSecureJSONData: map[string]string{
"tlsCACert": "root-cert-content",
"tlsClientCert": "client-cert-content",
"tlsClientKey": "client-key-content",
},
}
tlsConfig, err := tlsManager.getTLSSettings(dsInfo)
require.NoError(t, err)
assert.Equal(t, "require", tlsConfig.Mode)
assert.NotEmpty(t, tlsConfig.RootCertFile)
assert.NotEmpty(t, tlsConfig.CertFile)
assert.NotEmpty(t, tlsConfig.CertKeyFile)
// Cleanup temporary files
tlsManager.cleanupCertFiles(tlsConfig)
assert.NoFileExists(t, tlsConfig.RootCertFile)
assert.NoFileExists(t, tlsConfig.CertFile)
assert.NoFileExists(t, tlsConfig.CertKeyFile)
}
func resetWriteCertFile() {
writeCertFileCallNum = 0
writeCertFileFunc = writeCertFile
func TestTLSManager_CleanupCertFiles_FilePath(t *testing.T) {
logger := log.New()
tlsManager := newTLSManager(logger)
// Create temporary files for testing
rootCertFile, err := tlsManager.writeCertFile("root-*.crt", "root-cert-content")
require.NoError(t, err)
clientCertFile, err := tlsManager.writeCertFile("client-*.crt", "client-cert-content")
require.NoError(t, err)
clientKeyFile, err := tlsManager.writeCertFile("client-*.key", "client-key-content")
require.NoError(t, err)
// Simulate a configuration where the method is "file-path"
tlsConfig := tlsSettings{
ConfigurationMethod: "file-path",
RootCertFile: rootCertFile,
CertFile: clientCertFile,
CertKeyFile: clientKeyFile,
}
// Call cleanupCertFiles
tlsManager.cleanupCertFiles(tlsConfig)
// Verify the files are NOT deleted
assert.FileExists(t, rootCertFile, "Root certificate file should not be deleted")
assert.FileExists(t, clientCertFile, "Client certificate file should not be deleted")
assert.FileExists(t, clientKeyFile, "Client key file should not be deleted")
// Cleanup the files manually
err = os.Remove(rootCertFile)
require.NoError(t, err)
err = os.Remove(clientCertFile)
require.NoError(t, err)
err = os.Remove(clientKeyFile)
require.NoError(t, err)
}
func TestTLSManager_CreateCertFiles(t *testing.T) {
logger := log.New()
tlsManager := newTLSManager(logger)
dsInfo := sqleng.DataSourceInfo{
DecryptedSecureJSONData: map[string]string{
"tlsCACert": "root-cert-content",
"tlsClientCert": "client-cert-content",
"tlsClientKey": "client-key-content",
},
}
tlsConfig := tlsSettings{
ConfigurationMethod: "file-content",
}
err := tlsManager.createCertFiles(dsInfo, &tlsConfig)
require.NoError(t, err)
assert.FileExists(t, tlsConfig.RootCertFile)
assert.FileExists(t, tlsConfig.CertFile)
assert.FileExists(t, tlsConfig.CertKeyFile)
// Cleanup temporary files
tlsManager.cleanupCertFiles(tlsConfig)
assert.NoFileExists(t, tlsConfig.RootCertFile)
assert.NoFileExists(t, tlsConfig.CertFile)
assert.NoFileExists(t, tlsConfig.CertKeyFile)
}
func TestTLSManager_WriteCertFile(t *testing.T) {
logger := log.New()
tlsManager := newTLSManager(logger)
// Test writing a valid certificate file
filePath, err := tlsManager.writeCertFile("test-*.crt", "test-cert-content")
require.NoError(t, err)
assert.FileExists(t, filePath)
content, err := os.ReadFile(filepath.Clean(filePath))
require.NoError(t, err)
assert.Equal(t, "test-cert-content", string(content))
// Cleanup the file
err = os.Remove(filePath)
require.NoError(t, err)
assert.NoFileExists(t, filePath)
}
func TestTLSManager_CleanupCertFiles(t *testing.T) {
logger := log.New()
tlsManager := newTLSManager(logger)
// Create temporary files for testing
rootCertFile, err := tlsManager.writeCertFile("root-*.crt", "root-cert-content")
require.NoError(t, err)
clientCertFile, err := tlsManager.writeCertFile("client-*.crt", "client-cert-content")
require.NoError(t, err)
clientKeyFile, err := tlsManager.writeCertFile("client-*.key", "client-key-content")
require.NoError(t, err)
tlsConfig := tlsSettings{
ConfigurationMethod: "file-content",
RootCertFile: rootCertFile,
CertFile: clientCertFile,
CertKeyFile: clientKeyFile,
}
// Cleanup the files
tlsManager.cleanupCertFiles(tlsConfig)
// Verify the files are deleted
assert.NoFileExists(t, rootCertFile)
assert.NoFileExists(t, clientCertFile)
assert.NoFileExists(t, clientKeyFile)
}
@@ -259,6 +259,7 @@ const getStyles = (theme: GrafanaTheme2, headerLevels: number, headerHeight: num
display: 'flex',
flexDirection: 'column',
flexGrow: 1,
position: 'relative',
}),
pageContainerWithSidebar: css({
overflow: 'auto',
@@ -281,7 +282,9 @@ const getStyles = (theme: GrafanaTheme2, headerLevels: number, headerHeight: num
position: 'fixed !important' as 'fixed',
top: headerHeight,
bottom: 0,
zIndex: theme.zIndex.navbarFixed + 1,
// z-index needs to be above drawer mask (modal - 1 = 1059) so clicks on sidebar don't close drawer
// Using modal (1060) same as drawer - drawer will still appear on top as a modal overlay
zIndex: theme.zIndex.modal,
right: 0,
}),
};
@@ -13,6 +13,7 @@ import {
SceneObjectState,
SceneObjectStateChangedEvent,
SceneQueryRunner,
sceneGraph,
sceneUtils,
VizPanel,
isSceneObject,
@@ -21,6 +22,7 @@ import { Panel } from '@grafana/schema';
import { OptionFilter } from 'app/features/dashboard/components/PanelEditor/OptionsPaneOptions';
import { getLastUsedDatasourceFromStorage } from 'app/features/dashboard/utils/dashboard';
import { saveLibPanel } from 'app/features/library-panels/state/api';
import { getAllSuggestions } from 'app/features/panel/suggestions/getAllSuggestions';
import { DashboardEditActionEvent } from '../edit-pane/shared';
import { DashboardSceneChangeTracker } from '../saving/DashboardSceneChangeTracker';
@@ -29,6 +31,7 @@ import { UNCONFIGURED_PANEL_PLUGIN_ID } from '../scene/UnconfiguredPanel';
import { DashboardGridItem } from '../scene/layout-default/DashboardGridItem';
import { DashboardLayoutItem, isDashboardLayoutItem } from '../scene/types/DashboardLayoutItem';
import { vizPanelToPanel } from '../serialization/transformSceneToSaveModel';
import { PanelModelCompatibilityWrapper } from '../utils/PanelModelCompatibilityWrapper';
import {
activateSceneObjectAndParentTree,
getDashboardSceneFor,
@@ -84,7 +87,11 @@ export class PanelEditor extends SceneObjectBase<PanelEditorState> {
const panel = this.state.panelRef.resolve();
if (panel.state.pluginId === UNCONFIGURED_PANEL_PLUGIN_ID) {
panel.changePluginType('timeseries');
if (config.featureToggles.newVizSuggestions) {
this._autoSelectVisualization(panel);
} else {
panel.changePluginType('timeseries');
}
}
this._subs.add(
@@ -117,6 +124,31 @@ export class PanelEditor extends SceneObjectBase<PanelEditorState> {
};
}
private async _autoSelectVisualization(panel: VizPanel) {
const dataObject = sceneGraph.getData(panel);
this._subs.add(
dataObject.subscribeToState(async () => {
const { data } = dataObject.state;
const hasData = data && data.series && data.series.length > 0 && data.series.some((frame) => frame.length > 0);
if (hasData && panel.state.pluginId === UNCONFIGURED_PANEL_PLUGIN_ID) {
const panelModel = new PanelModelCompatibilityWrapper(panel);
const suggestions = await getAllSuggestions(data, panelModel);
if (suggestions.length > 0) {
const defaultFirstSuggestion = suggestions[0];
await panel.changePluginType(
defaultFirstSuggestion.pluginId,
defaultFirstSuggestion.options,
defaultFirstSuggestion.fieldConfig
);
}
}
})
);
}
private commitChanges() {
if (!this.state.isDirty && !this._changesHaveBeenMade) {
// Nothing to commit
@@ -228,12 +260,17 @@ export class PanelEditor extends SceneObjectBase<PanelEditorState> {
})
);
const isUnconfigured = Boolean(
config.featureToggles.newVizSuggestions && panel.state.pluginId === UNCONFIGURED_PANEL_PLUGIN_ID
);
// Setup options pane
this.setState({
optionsPane: new PanelOptionsPane({
panelRef: this.state.panelRef,
searchQuery: '',
listMode: OptionFilter.All,
isVizPickerOpen: isUnconfigured,
}),
isInitializing: false,
});
@@ -1,11 +1,13 @@
import { useCallback, useState } from 'react';
import { css } from '@emotion/css';
import { useCallback, useEffect, useState } from 'react';
import { PanelPlugin, PanelProps } from '@grafana/data';
import { CoreApp, GrafanaTheme2, PanelPlugin, PanelProps } from '@grafana/data';
import { Trans, t } from '@grafana/i18n';
import { locationService } from '@grafana/runtime';
import { config, locationService } from '@grafana/runtime';
import { sceneUtils } from '@grafana/scenes';
import { Box, Button, ButtonGroup, Dropdown, Menu, Stack } from '@grafana/ui';
import { Box, Button, ButtonGroup, Dropdown, Icon, Menu, Stack, Text, usePanelContext, useStyles2 } from '@grafana/ui';
import { NEW_PANEL_TITLE } from '../../dashboard/utils/dashboard';
import { DashboardInteractions } from '../utils/interactions';
import { findVizPanelByKey, getVizPanelKeyForPanelId } from '../utils/utils';
@@ -16,6 +18,8 @@ const UnconfiguredPanel = new PanelPlugin(UnconfiguredPanelComp);
function UnconfiguredPanelComp(props: PanelProps) {
const [isOpen, setIsOpen] = useState(false);
const panelContext = usePanelContext();
const styles = useStyles2(getStyles);
const onMenuClick = useCallback((isOpen: boolean) => {
setIsOpen(isOpen);
@@ -26,14 +30,15 @@ function UnconfiguredPanelComp(props: PanelProps) {
DashboardInteractions.panelActionClicked('configure', props.id, 'panel');
};
const onUseLibraryPanel = () => {
const dashboard = window.__grafanaSceneContext;
const dashboard = window.__grafanaSceneContext;
const panel =
dashboard instanceof DashboardScene ? findVizPanelByKey(dashboard, getVizPanelKeyForPanelId(props.id)) : null;
if (!(dashboard instanceof DashboardScene)) {
const onUseLibraryPanel = () => {
if (!dashboard || !(dashboard instanceof DashboardScene)) {
throw new Error('DashboardScene not found');
}
const panel = findVizPanelByKey(dashboard, getVizPanelKeyForPanelId(props.id));
if (!panel) {
throw new Error('Panel not found');
}
@@ -41,6 +46,18 @@ function UnconfiguredPanelComp(props: PanelProps) {
dashboard.onShowAddLibraryPanelDrawer(panel.getRef());
};
useEffect(() => {
if (!panel || !config.featureToggles.newVizSuggestions) {
return;
}
if (panelContext.app === CoreApp.PanelEditor) {
panel.setState({ title: '' });
} else if (!panel.state.title) {
panel.setState({ title: NEW_PANEL_TITLE });
}
}, [panel, panelContext.app]);
const MenuActions = () => (
<Menu>
<Menu.Item
@@ -56,6 +73,25 @@ function UnconfiguredPanelComp(props: PanelProps) {
</Menu>
);
const showEmptyState = config.featureToggles.newVizSuggestions && panelContext.app === CoreApp.PanelEditor;
if (showEmptyState) {
const defaultContent = (
<Trans i18nKey="dashboard.new-panel.empty-state-message">
Run a query to visualize it here or go to all visualizations to add other panel types
</Trans>
);
return (
<div className={styles.emptyStateWrapper}>
<Icon name="chart-line" size="xxxl" className={styles.emptyStateIcon} />
<Text element="p" textAlignment="center" color="secondary">
{defaultContent}
</Text>
</div>
);
}
return (
<Stack direction={'row'} alignItems={'center'} height={'100%'} justifyContent={'center'}>
<Box paddingBottom={2}>
@@ -79,3 +115,20 @@ sceneUtils.registerRuntimePanelPlugin({
pluginId: UNCONFIGURED_PANEL_PLUGIN_ID,
plugin: UnconfiguredPanel,
});
function getStyles(theme: GrafanaTheme2) {
return {
emptyStateWrapper: css({
display: 'flex',
flexDirection: 'column',
alignItems: 'center',
justifyContent: 'center',
height: '100%',
textAlign: 'center',
}),
emptyStateIcon: css({
color: theme.colors.text.secondary,
marginBottom: theme.spacing(2),
}),
};
}
@@ -96,6 +96,7 @@ function findVizPanelInternal(scene: SceneObject, key: string | undefined): VizP
return null;
}
export function findEditPanel(scene: SceneObject, key: string | undefined): VizPanel | null {
if (!key) {
return null;
@@ -258,8 +259,15 @@ export function getClosestVizPanel(sceneObject: SceneObject): VizPanel | null {
}
export function getDefaultVizPanel(): VizPanel {
const defaultPluginId = config.featureToggles.dashboardNewLayouts ? UNCONFIGURED_PANEL_PLUGIN_ID : 'timeseries';
const newPanelTitle = t('dashboard.new-panel-title', 'New panel');
const defaultPluginId =
config.featureToggles.dashboardNewLayouts || config.featureToggles.newVizSuggestions
? UNCONFIGURED_PANEL_PLUGIN_ID
: 'timeseries';
const newPanelTitle =
config.featureToggles.newVizSuggestions && defaultPluginId === UNCONFIGURED_PANEL_PLUGIN_ID
? ''
: t('dashboard.new-panel-title', 'New panel');
return new VizPanel({
title: newPanelTitle,
@@ -128,6 +128,7 @@ export class LogListModel implements LogRowModel {
clone._wrapLogMessage = true;
clone._body = undefined;
clone._highlightTokens = undefined;
clone.collapsed = false;
return clone;
}
@@ -5,7 +5,8 @@ import AutoSizer from 'react-virtualized-auto-sizer';
import { GrafanaTheme2, PanelData, PanelModel, PanelPluginVisualizationSuggestion } from '@grafana/data';
import { Trans } from '@grafana/i18n';
import { useStyles2 } from '@grafana/ui';
import { config } from '@grafana/runtime';
import { Icon, Text, useStyles2 } from '@grafana/ui';
import { getAllSuggestions } from '../../suggestions/getAllSuggestions';
@@ -31,6 +32,21 @@ export function VisualizationSuggestions({ searchQuery, onChange, data, panel, t
return result;
}, [searchQuery, suggestions, trackSearch]);
const hasData = data?.series && data.series.length > 0 && !data.series.every((frame) => frame.length === 0);
if (config.featureToggles.newVizSuggestions && !hasData && !searchQuery) {
return (
<div className={styles.emptyStateWrapper}>
<Icon name="chart-line" size="xxxl" className={styles.emptyStateIcon} />
<Text element="p" textAlignment="center" color="secondary">
<Trans i18nKey="dashboard.new-panel.suggestions.empty-state-message">
Run a query to start seeing suggested visualizations
</Trans>
</Text>
</div>
);
}
return (
// This div is needed in some places to make AutoSizer work
<div>
@@ -116,5 +132,18 @@ const getStyles = (theme: GrafanaTheme2) => {
marginBottom: theme.spacing(1),
justifyContent: 'space-evenly',
}),
emptyStateWrapper: css({
display: 'flex',
flexDirection: 'column',
alignItems: 'center',
justifyContent: 'center',
padding: theme.spacing(4),
textAlign: 'center',
minHeight: '200px',
}),
emptyStateIcon: css({
color: theme.colors.text.secondary,
marginBottom: theme.spacing(2),
}),
};
};
@@ -1,9 +1,10 @@
import { CustomVariableModel } from '@grafana/data';
import { AzureLogsQuery, AzureQueryType, AzureTracesQuery } from '../dataquery.gen';
import { Context, createContext } from '../mocks/datasource';
import createMockQuery from '../mocks/query';
import { singleVariable } from '../mocks/variables';
import { AzureLogsQuery, AzureMonitorQuery, AzureQueryType, AzureTracesQuery } from '../types/query';
import { AzureMonitorQuery } from '../types/query';
import AzureLogAnalyticsDatasource from './azure_log_analytics_datasource';
import FakeSchemaData from './mocks/schema';
@@ -6,7 +6,8 @@ import { DataSourceWithBackend, getTemplateSrv, TemplateSrv } from '@grafana/run
import ResponseParser from '../azure_monitor/response_parser';
import { getCredentials } from '../credentials';
import { AzureMonitorQuery, AzureQueryType } from '../types/query';
import { AzureQueryType } from '../dataquery.gen';
import { AzureMonitorQuery } from '../types/query';
import {
AzureMonitorDataSourceJsonData,
AzureMonitorDataSourceInstanceSettings,
@@ -5,8 +5,9 @@ import { ScopedVars } from '@grafana/data';
import { DataSourceWithBackend, getTemplateSrv, TemplateSrv } from '@grafana/runtime';
import { getCredentials } from '../credentials';
import { AzureMetricQuery, AzureQueryType } from '../dataquery.gen';
import TimegrainConverter from '../time_grain_converter';
import { AzureMetricQuery, AzureMonitorQuery, AzureQueryType } from '../types/query';
import { AzureMonitorQuery } from '../types/query';
import {
AzureAPIResponse,
AzureMonitorDataSourceInstanceSettings,
@@ -1,6 +1,6 @@
import { TemplateSrv } from '@grafana/runtime';
import { AzureMonitorResource } from '../types/query';
import { AzureMonitorResource } from '../dataquery.gen';
import { GetMetricNamespacesQuery, GetMetricNamesQuery } from '../types/types';
export default class UrlBuilder {
@@ -2,12 +2,12 @@ import { get, set } from 'lodash';
import { CustomVariableModel } from '@grafana/data';
import { AzureQueryType } from '../dataquery.gen';
import { Context, createContext } from '../mocks/datasource';
import { createMockInstanceSetttings } from '../mocks/instanceSettings';
import createMockQuery from '../mocks/query';
import { createTemplateVariables } from '../mocks/utils';
import { multiVariable, singleVariable, subscriptionsVariable } from '../mocks/variables';
import { AzureQueryType } from '../types/query';
import AzureResourceGraphDatasource from './azure_resource_graph_datasource';
@@ -4,9 +4,9 @@ import { ScopedVars } from '@grafana/data';
import { DataSourceWithBackend, getTemplateSrv, TemplateSrv } from '@grafana/runtime';
import { resourceTypes } from '../azureMetadata/resourceTypes';
import { ARGScope } from '../dataquery.gen';
import { ARGScope, AzureQueryType } from '../dataquery.gen';
import { createFilter } from '../resourcePicker/resourcePickerData';
import { AzureMonitorQuery, AzureQueryType } from '../types/query';
import { AzureMonitorQuery } from '../types/query';
import {
AzureGetResourceNamesQuery,
AzureGraphResponse,

Some files were not shown because too many files have changed in this diff Show More