Commit Graph
21 Commits
Author SHA1 Message Date
Alexander Akhmetov 100528e274 Alerting: Support retry with backoff in alert rule evaluation (#99710) 2025-09-04 13:56:03 +02:00
Alexander Akhmetov 89d6756c67 Alerting: Filter out private labels before writing recording rules (#109295) 2025-08-07 17:25:12 +02:00
Yuri TseretyanandWilliam Wernert dc0083d879 Alerting: Sequential evaluation of rules in group (#98829)
* introduce RulesGroupComparer

* extract runJob method

* implement sequential evaluation

* Make sequence building testable & add comments

* Also run callback in recording rules + add tests

* Improve tests

* Address PR comments

---------

Co-authored-by: William Wernert <william.wernert@grafana.com>
2025-04-02 23:10:32 +03:00
Steve Simpson b7dcfcedcb Alerting: Extend recording rule definitions/interfaces with data source. (#101678)
Extend the recording rule definition to include the target data source, allowing
configuration of where the output of the recording rule is written to. Also
extends the relevant interfaces in preparation for the next set of changes.
2025-03-06 14:09:17 +01:00
Yuri Tseretyan 32fde6dba4 Alerting: Update scheduler to provide full specification to rule update channel (#101375)
update scheduler's aler rule to accept regular Evaluation in update channel

This makes it accept the full rule definition, which is required in reset state.
2025-02-26 14:39:39 -05:00
Alexander Akhmetov a28328d764 Alerting: Call the deletion reason provider even if the rule is no longer scheduled (#99571)
Alerting: Call the deletion reason provider even if the rule is not scheduled anymore
2025-01-28 11:34:26 +01:00
Alexander Weaver 393faa8732 Alerting: Move rule evaluation status logic out of prometheus API and into scheduler (#89141)
* Add health fields to rules and an aggregator method to the scheduler

* Move health, last error, and last eval time in together to minimize state processing

* Wire up a readonly scheduler to prom api

* Extract to exported function

* Use health in api_prometheus and fix up tests

* Rename health struct to status

* Fix tests one more time

* Several new tests

* Handle inactive rules

* Push state mapping into state manager

* rename to StatusReader

* Rectify cyclo complexity rebase

* Convert existing package local status implementation to models one

* fix tests

* undo RuleDefs rename
2024-09-30 16:52:49 -05:00
Alexander Weaver ac5ebe6e4d Alerting: Add enablement flag for recording rules (#92032)
* Add enablement flag

* Disable if toggle not enabled
2024-08-19 12:01:00 -05:00
Alexander Weaver 34ab5fe1f3 Alerting: Restart rule routines if the type changes (#90867)
* Restart when types change

* Wire up test hooks correctly

* testing
2024-08-14 14:57:47 -05:00
Alexander Weaver 72ecde5045 Alerting: Make orgID a direct arg of writer interface (#91422)
make orgID a direct arg of writer interface
2024-08-02 09:37:28 -05:00
Alexander Weaver 88ed77e7e8 Alerting: More graceful handling of NoData in recording rules (#90312)
* Handle NoData as its own case

* Debug

* Scalars parseable by CollectionReader

* fix linter

* Orgit add pkg/*git add pkg/* not and
2024-07-17 15:24:03 -05:00
Yuri Tseretyan c3b9c9b239 Alerting: Send information about alert rule to data source in headers (#90344)
* add support of metadata to condition and adding it to request headers
* support for additional metadata when condition is built
* add additionall context to conditions: source and folder title
* add version
* use percent-encoding for header values
2024-07-17 22:55:12 +03:00
Alexander Weaver ab32183e18 Alerting: Track recording rule health and last eval info ephemerally (#90247)
* Track health and last eval info

* Read method for status

* Minor tests
2024-07-11 14:05:09 -05:00
Yuri Tseretyan c3b5cabb14 Alerting: Refactor scheduler's rule evaluator to store rule key (#89925) 2024-07-01 16:43:23 -04:00
William Wernert fcfa89f864 Alerting: Implement Prometheus remote write for recording rules (#89189)
* Fix timestamp recorded by rule

* Implement prometheus remote write

* Create http client instead of transport

* Address PR comments

* Remove status code label
2024-06-25 17:23:42 +03:00
William Wernert c62cc25513 Alerting: Configure recording rule writer from config.ini (#89056) 2024-06-12 16:04:46 -04:00
Alexander Weaver d004f8a98d Alerting: Recording rules understands errors embedded in dataframes (#88946)
* Make MakeDependencyError public for tests in another package

* Create tests for errors in eval results

* Extract logic to pull frame errors out into exported function

* Maybe we can drop cyclomatic complexity lint suppression now?

* extract frame errors and fail recording rules if frames contain error

* Fix up retry logic to actually work

* Do not retry non retryable errors
2024-06-11 10:37:10 -05:00
Alexander Weaver 58fdb24b0b Alerting: Recording rules appear as type=recording in Prometheus API + better abstraction for type (#88805)
* Wire status through to prom API

* Regenerate swagger
2024-06-07 11:24:06 -05:00
William Wernert 5de7d4d06d Alerting: Create writer interface for recording rules (#88459)
* Create writer interface for recording rules

Also create fake impl + use it for stub in scheduler
2024-05-29 22:38:33 +03:00
Alexander WeaverandSteve Simpson b926b6336d Alerting: Scheduled recording rules execute their queries (#88309)
* Basic eval flow

* Wiring-up

* fix

* Extend todo

* Start with tests

* Include some relevant tests, skip ones that seem to have timing-based race conditions

* Some tests, touch up linter and todo

* Solve TODO

* Add tracing

* Tests to make sure an eval went through

* Wire up feature toggles

* Update pkg/services/ngalert/schedule/recording_rule.go

Co-authored-by: Steve Simpson <steve.simpson@grafana.com>

* Update pkg/services/ngalert/schedule/recording_rule_test.go

Co-authored-by: Steve Simpson <steve.simpson@grafana.com>

* Update pkg/services/ngalert/schedule/recording_rule_test.go

Co-authored-by: Steve Simpson <steve.simpson@grafana.com>

* Update pkg/services/ngalert/schedule/recording_rule_test.go

Co-authored-by: Steve Simpson <steve.simpson@grafana.com>

---------

Co-authored-by: Steve Simpson <steve.simpson@grafana.com>
2024-05-28 10:59:21 -05:00
Alexander Weaver 89b54d06e9 Alerting: Schedule a shim implementation for recording rules (#87939)
* Add shim rule implementation for recording rules

* Give ruleFactory access to the original rule definition

* Schedule shim implementation if the rule is a recording rule

* Fix or suppress linter

* Fix nolint
2024-05-21 16:42:58 -05:00