Commit Graph

27 Commits

Author SHA1 Message Date
Grot (@grafanabot) 6c512e1414 Alerting: Expand {{$labels.xyz}} template in labels and annotations (#35159) (#35208)
* Alerting: Expand `{{$labels.xyz}}` template in labels and annotations

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>

* Fix annotation not updating for same alert

Signed-off-by: Ganesh Vernekar <ganeshvern@gmail.com>
(cherry picked from commit 8417088969)

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-06-04 11:05:19 +05:30
Grot (@grafanabot) d99847423b set state correctly and test (#34680) (#34770)
(cherry picked from commit 20d356947c)

Co-authored-by: David Parrott <stomp.box.yo@gmail.com>
2021-05-26 15:04:40 -07:00
David Parrott 7a83d1f9ff Alerting resend delay for sending to notifiers (#34312)
* adds resend delay to avoid saturating notifier

* correct method signatures

* pr feedback
2021-05-19 22:15:09 +02:00
David Parrott 25485100b0 Alerting: Trim results when at processing instead of on ticker (#34248)
* Trim results when at processing instead of on ticker

* User RWMutex correctly

* remove comment
2021-05-18 10:56:14 -07:00
David Parrott bbb7bbf891 Alerting: Remove back end logic for supporting KeepLastState (#34242)
* Removed back end logic for supporting KeepLastState

* Map keep_state correctly in migrations
2021-05-18 10:55:43 -07:00
Kyle Brandt 63b2dd06a5 Alerting: Set "value" with evalmatches in G Managed (#34075)
When, and currently only when using a classic condition, evaluation information is added (which is like the EvalMatches from dashboard alerting).

This is returned via the API and can be included in notifications by reading the `__value__` label attached `.Alerts` in the template. It is a string.
2021-05-18 09:12:39 -04:00
Owen Diehl 1367f7171e Alerting/ruler metrics (#34144)
* adds active configurations metric

* rule evaluation metrics

* ruler metrics

* pr feedback
2021-05-14 16:13:44 -04:00
Kyle Brandt babb17afd6 Alerting/Chore: Move tests from tests package (#34059)
Instead put in package folder but with package name suffixed with _test
This enables code coverage within the pkg while still allow the tests to operate from external to package perspective (only exported things).
2021-05-13 10:05:33 -04:00
Kyle Brandt fae093bbe2 Alerting: Fix state cache getOrCreate panic (#33777) 2021-05-06 14:35:52 +02:00
David Parrott b1a8c67689 Alerting return evaluation errors to /rules (#33663)
* Set and return errors produced by evaluation results

* test fixup
2021-05-04 13:08:12 -04:00
David Parrott 39099bf3c0 Alerting nested state cache (#33666)
* nest cache by orgID, ruleUID, stateID

* update accessors to use new cache structure

* test and linter fixup

* fix panic

Co-authored-by: Kyle Brandt <kyle@grafana.com>

* add comment to identify what's going on with nested maps in cache

Co-authored-by: Kyle Brandt <kyle@grafana.com>
2021-05-04 09:57:50 -07:00
Kyle Brandt 48358efc13 Alerting: remove State cache entries on Ruler Delete (#33638)
for https://github.com/grafana/alerting-squad/issues/133
2021-05-03 14:01:33 -04:00
Owen Diehl 070627d11e better handle metrics for state transitions (#33648) 2021-05-03 11:57:24 -04:00
Kyle Brandt 7823842c5d Alerting: Load annotations from rule into State cache (#33542)
for https://github.com/grafana/alerting-squad/issues/127
2021-04-30 20:23:12 +02:00
Owen Diehl 5e48b54549 Alerting/metrics (#33547)
* moves alerting metrics to their own pkg

* adds grafana_alerting_alerts (by state) metric

* alerts_received_{total,invalid}

* embed alertmanager alerting struct in ng metrics & remove duplicated notification metrics (already embed alertmanager notifier metrics)

* use silence metrics from alertmanager lib

* fix - manager has metrics

* updates ngalert tests

* comment lint
Signed-off-by: Owen Diehl <ow.diehl@gmail.com>

* cleaner prom registry code

* removes ngalert global metrics

* new registry use in all tests

* ngalert metrics impl service, hack testinfra code to prevent duplicate metric registrations

* nilmetrics unexported
2021-04-30 12:28:06 -04:00
Sofia Papagiannaki 1e380e869e [Alerting]: some fixes (#33538)
* Fix fialure when adding state annotations

* Fix get org rules API

Do not fail response if user has no access to view a namespace.
Do not include the namespace in the response instead.

* lint
2021-04-29 19:15:15 +03:00
Kyle Brandt 914443c816 Alerting: Fix state cache id duplication (#33480) 2021-04-28 11:42:19 -04:00
David Parrott 788bc2a793 Alerting: refactor state tracker (#33292)
* set processing time

* merge labels and set on response

* use state cache for adding alerts to rules

* minor cleanup

* add support for NoData and Error results

* rename test

* bring in changes from other PRs tha have been merged

* pr feedback

* add integration test

* close state tracker cleanup on context.Done

* fixup test

* rename state tracker

* set EvaluationDuration on Result

* default labels set as constants

* separate cache and state from manager

* use RWMutex in cache
2021-04-23 21:32:25 +02:00
David Parrott ca79206498 Alerting: Handle NoData and Error evaluation results (#33194)
* set processing time

* merge labels and set on response

* use state cache for adding alerts to rules

* minor cleanup

* add support for NoData and Error results

* rename test

* bring in changes from other PRs tha have been merged

* pr feedback

* add integration test

* close state tracker cleanup on context.Done

* fixup test

* not those annotations
2021-04-23 20:47:52 +02:00
gotjosh de0802cf3b Alerting: Fixes the integration test currently failing at master (#33233)
* Alerting: Fixes the integration test currently failing at master

* Skip the state tracker test for now
2021-04-21 14:57:17 -04:00
David Parrott 4be1d84f23 Alerting: Enhancements to /rules (#33085)
* set processing time

* merge labels and set on response

* use state cache for adding alerts to rules

* minor cleanup

* pr feedback

* Do not initialize mutex unnecessarily

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>

* linter

Co-authored-by: Ganesh Vernekar <15064823+codesome@users.noreply.github.com>
2021-04-21 09:30:03 -07:00
David Parrott 555da77527 Dparrott/labels on alert rule (#33057)
* move state tracker tests to /tests

* set default labels on alerts

* handle empty labels in result.Instance

* create annotation on transition to alerting state
2021-04-16 15:11:40 +02:00
David Parrott 567a6a09bd Alerting: Return RuleResponse for api/prometheus/grafana/api/v1/rules (#32919)
* Return RuleResponse for api/prometheus/grafana/api/v1/rules

* change TODO to note

Co-authored-by: gotjosh <josue@grafana.com>

* pr feedback

* test fixup

Co-authored-by: gotjosh <josue@grafana.com>
2021-04-13 17:38:09 -04:00
David Parrott c0d83fc01e Alerting: Return cached alerts for prometheus/api/v1/alerts (#32654)
* Return cached alerts for prometheus/api/v1/alerts

* Return not implemented for /prometheus/grafana/api/v1/rules

* Set StartsAt for already alerting states

* Fix tests
2021-04-05 15:05:39 -07:00
David Parrott 2a8446e435 Alerting: Persist alerts on evaluation and shutdown. Warm cache from DB on startup (#32576)
* Initial commit for state tracking

* basic state transition logic and tests

* constructor. test and interface fixup

* use new sig for sch.definitionRoutine()

* test fixup

* make the linter happy

* more minor linting cleanup

* Alerting: Send alerts from state tracker to notifier

* Add evaluation time and test

Add evaluation time and test

* Add cleanup routine and logging

* Pull in compact.go and reconcile differences

* Save alert transitions and save all state on shutdown

* pr feedback

* WIP

* WIP

* Persist alerts on evaluation and shutdown. Warm cache on startup

* Filter non-firing alerts before sending to notifier

Co-authored-by: Josue Abreu <josue@grafana.com>
2021-04-02 08:11:33 -07:00
David Parrott b1cb74c0c9 Alerting: Send alerts from state tracker to notifier, logging, and cleanup task (#32333)
* Initial commit for state tracking

* basic state transition logic and tests

* constructor. test and interface fixup

* use new sig for sch.definitionRoutine()

* test fixup

* make the linter happy

* more minor linting cleanup

* Alerting: Send alerts from state tracker to notifier

* Add evaluation time and test

Add evaluation time and test

* Add cleanup routine and logging

* Pull in compact.go and reconcile differences

* pr feedback

* pr feedback

Pull in compact.go and reconcile differences

Co-authored-by: Josue Abreu <josue@grafana.com>
2021-03-30 09:37:56 -07:00
David Parrott d33a77a67f Alerting: add state tracker to alerting evaluation (#32298)
* Initial commit for state tracking

* basic state transition logic and tests

* constructor. test and interface fixup

* use new sig for sch.definitionRoutine()

* test fixup

* make the linter happy

* more minor linting cleanup
2021-03-24 15:34:18 -07:00