Commit Graph

191 Commits

Author SHA1 Message Date
Sam Jewell
21b9d45ca6 SQL Expressions: Add CASE/WHEN nodes and fixes (and test) for functions just added to allowlist (#102040)
* SQL Expressions: Add CASE/WHEN SQL nodes to allowlist

* Fixed and test for functions added in #102011

* Add remaining functions to the test-case

These are mostly aliases, so the LLM chose to omit them originally.
But adding now for completeness

* Fix ordering of allowed nodes
2025-03-12 15:57:50 +00:00
Sam Jewell
4bd5f29e05 SQL Expressions: Allow more functions (#102011)
I used Cursor and Claude 3.7 thinking to expand the list of functions
we allow. Specifically I needed `abs`.

TODO: Check each of these in the official MySQL docs to ensure they
are all safe:
eg. this doc: https://dev.mysql.com/doc/refman/8.4/en/flow-control-functions.html#operator_case
2025-03-12 11:39:48 +00:00
Sam Jewell
7a3415148e SQL Expressions: Add cell-limit for input dataframes (#101700)
* expr: Add row limit to SQL expressions

Adds a configurable row limit to SQL expressions to prevent memory issues with large
result sets. The limit is configured via the `sql_expression_row_limit` setting in
the `[expressions]` section of grafana.ini, with a default of 100,000 rows.

The limit is enforced by checking the total number of rows across all input tables
before executing the SQL query. If the total exceeds the limit, the query fails
with an error message indicating the limit was exceeded.

* revert addition of newline

* Switch to table-driven tests

* Remove single-frame test-cases.

We only need to test for the multi frame case. Single frame is a subset
of the multi-frame case

* Add helper function

Simplify the way tests are set up and written

* Support convention, that limit: 0 is no limit

* Set the row-limit in one place only

* Update default limit to 20k rows

As per some discussion here:
https://raintank-corp.slack.com/archives/C071A5XCFST/p1741611647001369?thread_ts=1740047619.804869&cid=C071A5XCFST

* Test row-limit is applied from config

Make sure we protect this from regressions

This is perhaps a brittle test, somewhat coupled to the code here. But
it's good enough to prevent regressions at least.

* Add public documentation for the limit

* Limit total number of cells instead of rows

* Use named-return for totalRows

As @kylebrandt requested during review of #101700

* Leave DF cells as zero values during limits tests

When testing the cell limit we don't interact with the cell values at
all, so we leave them at their zero values both to speed up tests, and
to simplify and clarify that their values aren't used.

* Set SQLCmd limit at object creation - don't mutate

* Test that SQL node receives limit when built

And that it receives it from the Grafana config

* Improve TODO message for new Expression Parser

* Fix failing test by always creating config on the Service
2025-03-11 17:14:33 +00:00
Sam Jewell
0eafd0641f SQL Expressions: Simplify test code (#101718)
Simplify, by retaining quotes when escaping query

We also don't need to convert to string when passing to fmt.Sprintf
a slice of Bytes is accepted by Sprintf instead.
2025-03-06 17:02:31 +00:00
Matheus Macabu
3539764008 Hackaton: Add more unit tests, take 3 (#101525)
* serviceaccounts/secretscan: test Service more thoroughly

* middleware/cookies: add tests for CookieOptions

* anonymous/anonimpl: cover a couple more methods

* components/imguploader: Implement WebDAV integration tests

* components/apikeygen: also check IsValid method

* bus: cover invalid callback signature cases

* cloudmigration/objectstorage: add basic unit tests

* login/social/connectors: add test case for GitHub OAuth fetch emails+orgs

* expr/classic: cover more evaluator types in tests
2025-03-05 08:00:12 +01:00
Paulo Dias
2d2e595555 Alerting: Add multiple threshold operators (#99516)
The following operators are being added:
- Equal
- Not Equal
- Greater or Equal
- Less or Equal
- Within Range Inclusive
- Outside Range Inclusive
2025-02-21 19:11:16 +02:00
Kyle Brandt
4d7b9a3c77 SQL Expressions: Improve response to frame conversion handling (#100625)
- use types.Convert to get a more normalized type returned from GMS. After syncing with GMS maintainers, the underlying type of the row could be different things (e.g. int when unit32, so this calls the Convert method from the GMS types library to normalize the interface.
- Clean up fieldValFromRow more
- Use IsText to capture different string types
- Add more types to test, also update test to use same cmp.Diff method as others

---------

Co-authored-by: Sam Jewell <2903904+samjewell@users.noreply.github.com>
2025-02-14 17:21:08 +02:00
Kyle Brandt
c291ec7ba9 SQL Expressions: Include SQL Parser/Syntax error in the public message (#100725)
fixes #100721
2025-02-14 09:09:21 -05:00
Kyle Brandt
f7588376df SQL Expressions: (Chore) Update code for latest plugin-sdk data pkg (#100425)
Use new NilAt and SetRefId methods
2025-02-11 15:15:54 +00:00
Kyle Brandt
6dc98dbbcc SQL Expressions: Add str_to_date function and unskip test (#100226) 2025-02-07 11:51:55 +02:00
Kyle Brandt
d196b789e2 SQL Expressions: Add more SQLNodes and funcs to allow list (#100227)
sql_expr: Add more Nodes and funcs to allow list
2025-02-07 09:44:17 +00:00
Kyle Brandt
d64f41afdc SQL Expressions: Re-implement feature using go-mysql-server (#99521)
* Under feature flag `sqlExpressions` and is experimental
* Excluded from arm32
* Will not work with the Query Service yet
* Does not have limits in place yet
* Does not working with alerting yet
* Currently requires "prepare time series" Transform for time series viz
 
---------

Co-authored-by: Sam Jewell <sam.jewell@grafana.com>
2025-02-06 07:27:28 -05:00
Karl Persson
76f052e8de Requester: Remove duplicated function (#97038)
* Remove duplicated function

* Remove GetDisplayName from interface

* Use GetName
2024-11-26 15:29:31 +01:00
Sam Jewell
1915efbc95 SQL Expressions: Simplify where code is being stubbed out (#96227)
* Rename from DuckDB

* Remove unused function
2024-11-18 12:05:08 +00:00
Sam Jewell
ea71201ddc ServerSideExpressions: Disable SQL Expressions to prevent RCE and LFI vulnerability (#94942)
* disable sql expressions

remove duckdb ref

* Run `make update-workspace`

---------

Co-authored-by: Scott Lepper <scott.lepper@gmail.com>
2024-10-18 11:28:26 +01:00
Gábor Farkas
cbe1e7d63f expr: add explicit strict-mode const (#94838)
expr: add explicit strict-mode value
2024-10-18 10:46:27 +02:00
Gábor Farkas
34bb959835 expr: improved test (#94837) 2024-10-18 10:31:58 +02:00
Gábor Farkas
2cea7af2f5 datasources: querier: handle reducer-settings-mode correctly (#94595)
* datasources: querier: handle reducer-settings-mode correctly

* Try to fix linting error

"unnecessary leading newline (whitespace)"

---------

Co-authored-by: Sam Jewell <sam.jewell@grafana.com>
2024-10-17 08:38:14 +02:00
Andrew Hackmann
3df62c28bb InfluxDB: Add Influx to check if series need to be fixed labels (#93299)
add influx to check if series need to be fixed labels
2024-09-13 14:05:15 -05:00
Dave Henderson
df3d8915ba Chore: Bump Go to 1.23.0 (#92105)
* chore: Bump Go to 1.23.0

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* update swagger files

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore: update .bingo/README.md formatting to satisfy prettier

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore(lint): Fix new lint errors found by golangci-lint 1.60.1 and Go 1.23

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* keep golden file

* update openapi

* add name to expected output

* chore(lint): rearrange imports to a sensible order

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-08-21 11:40:42 -04:00
Lu Shueh Chou
126169f3ca Alerting: Add LogicOr operation option (#89258)
---------

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
2024-08-16 11:02:14 -04:00
Alexander Akhmetov
a32854549c Alerting: Support median in reduce expressions (#91119)
* Alerting: support median in reduce expressions
2024-08-01 15:04:17 +02:00
Yuri Tseretyan
9c05b30489 Chore: Add more logs and tracing to hysteresis flows (#90369) 2024-07-15 13:38:20 -04:00
Fabrizio
612ebf5508 Dynatrace: Add to list of DS with custom label logic (#90258) 2024-07-15 10:18:54 +02:00
Alexander Weaver
b3d863c11e Expr: Make math expression scalars compliant with dataplane contract (#90350)
* Specify frame type for scalars arbitrarily

* Update alerting tests which assert on entire dataframe fields
2024-07-12 12:52:19 -05:00
Dave Henderson
6262c56132 chore(perf): Pre-allocate where possible (enable prealloc linter) (#88952)
* chore(perf): Pre-allocate where possible (enable prealloc linter)

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* fix TestAlertManagers_buildRedactedAMs

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* prealloc a slice that appeared after rebase

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-06-14 14:16:36 -04:00
Kyle Brandt
82aa000e9d SSE: (Chore) Update log line with context (trace) (#89172)
Improves log line to help with debugging in Server Side Expressions. In particular, the traceId, datasourceType, and datasourceUid will now be included.
2024-06-13 10:58:39 -04:00
Ryan McKinley
99d8025829 Chore: Move identity and errutil to apimachinery module (#89116) 2024-06-13 07:11:35 +03:00
Alexander Weaver
d004f8a98d Alerting: Recording rules understands errors embedded in dataframes (#88946)
* Make MakeDependencyError public for tests in another package

* Create tests for errors in eval results

* Extract logic to pull frame errors out into exported function

* Maybe we can drop cyclomatic complexity lint suppression now?

* extract frame errors and fail recording rules if frames contain error

* Fix up retry logic to actually work

* Do not retry non retryable errors
2024-06-11 10:37:10 -05:00
Marcus Efraimsson
2403665998 Chore: Remove legacydata package (#88790)
* Chore: Remove legacydata package

* remove unused dependency

* sdk v0.234.0
2024-06-05 22:31:06 +02:00
Yuri Tseretyan
043e19afb5 SSE: Fix threshold unmarshal to avoid panic (#88521) 2024-06-03 16:16:13 -04:00
Ryan McKinley
ffc2702552 Plugins: Support Admission validation hooks (#87718) 2024-05-24 18:45:16 +03:00
Ryan McKinley
42b0f802de QueryTypes: Add feature toggle to show query types in datasource apiservers (#88213)
* initial attempt

* show query types

* show query types

* with formatting

* with formatting

* more cleanup

* add feature toggle

* fix build
2024-05-23 18:46:28 +02:00
Scott Lepper
cac40795a5 sql expressions - logging (#88193)
* sql expressions - logging
2024-05-22 17:18:46 -04:00
Scott Lepper
93acc0c932 sqlexpressions - fix escape quote (#87921) 2024-05-15 10:55:58 -04:00
Scott Lepper
14a814a280 [sql expressions] fix: use ast to read tables (#87867)
* [sql expressions] fix: use ast to read tables

* can't run tests during ci yet.  need to install duckdb

* skip for now.  need duckdb cli
2024-05-14 17:05:29 -04:00
Alexander Weaver
6c47968f6c Alerting: Do not retry rule evaluations with "input data must be a wide series but got type long" style errors (#87343)
add typed error for series must be wide, do not retry
2024-05-07 11:31:07 -05:00
Scott Lepper
1a2bbd61fd sql expressions: improve parser (#87277)
sql expressions: improve parser
2024-05-03 08:08:07 -04:00
Scott Lepper
cbe3e3a5e0 fix: sql expressions parser with join (#87263) 2024-05-02 14:48:05 -04:00
Scott Lepper
4fd2cb6014 fix: sql expressions - sql parser table name case (#87196)
* fix: sql parser table name case
2024-05-02 08:43:20 -04:00
Yuri Tseretyan
e9d6135e33 SSE: Threshold expression to use simple functions (#86062)
* replace math expression with predicates
2024-04-16 13:35:41 -04:00
Nick Richmond
d3fee607e2 Expressions: sort numeric metrics behind feature toggle (#85911)
* feat: sort numeric metrics behind feature toggle

* chore: upgrade `dataplane/sdata` to latest tag

* chore: `go work sync`
2024-04-16 10:52:47 -04:00
Faye Lin
a94ab259c6 Expression: Fix a bug of the display name of the threshold expression result (#85912)
* replace ReferenceVar with RefID when creating the math command for the ThresholdCommand

* Fix the Hysteresis test
2024-04-12 17:46:14 +02:00
Dave Henderson
5687243d0b Feature Flags: use FeatureToggles interface where possible (#85131)
* Feature Flags: use FeatureToggles interface where possible

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Replace TestFeatureToggles with existing WithFeatures

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-04-04 12:22:31 -04:00
Ryan McKinley
4cda34ff7d Expressions: Create json schema for query types (#84032) 2024-03-26 06:58:56 +02:00
Yuri Tseretyan
9dc4221508 Alerting: Log expression command types during evaluation (#84614) 2024-03-19 10:00:03 -04:00
carrychair
0913324668 Chore: Remove repetitive words (#84132)
remove repetitive words

Signed-off-by: carrychair <linghuchong404@gmail.com>
2024-03-11 08:55:18 -04:00
Ryan McKinley
d82f3be6f7 QueryService: Use types from sdk (#84029) 2024-03-08 18:12:59 +02:00
Ryan McKinley
3036b50df3 Expressions: expose ConvertDataFramesToResults (#83805) 2024-03-04 18:22:56 +02:00
Ryan McKinley
5f6bf93dd5 Expressions: Use enumerations rather than strings (#83741) 2024-03-01 19:38:32 +02:00