* add function to convert StateTransition to LokiEntry
* add QueryResultBuilder
* update backtesting to produce result similar to historian
* make shouldRecord public
* filter out noop transitions
* add experimental front-end
* add new fields
* move conversion of api model to AlertRule to validation
* add extra labels
* calculate tick timestamp using the same logic as in scheduler
* implement correct logic of calculating first evaluation timestamp
* add uid, group and folder uid they are needed for jitter strategy
* add JitterOffsetInDuration and JitterStrategy.String()
* add config `backtesting_max_evaluations` to [unified_alerting] (not documented for now)
* remove obsolete tests
* elevate permisisons for backtesting endpoint
* move backtesting to separate dir
* Alerting: Improve ASH Loki query efficiency by including folderUID
Previously, the folderUID label was only included when ruleUID was not specified
and the user did not have full alert rule read permissions.
To improve ASH Loki query efficiency, this PR includes the folderUID in the ASH
Loki query when ruleUID is specified, even if the user has full alert rule read
permissions.
Some non-obvious considerations:
- The naive implementation of just including the current folder UID would have
the unintended side-effect of no longer returning history after a rule is moved
between folders.
- The previous implementation made the trade-off of only checking RBAC on the
current folder, including any history from old folders that may exist.
To solve both of the above, we make an extra query to the database to check the
alert rule's previous versions so we can include any old folderUIDs, checking
RBAC at the same time.
The querying and inclusion of history from old folders is done best-effort, any
issues that might arise are logged and ignored so as not to prevent the current
folder history.
* Fix merge conflicts
* Reduce scanning on GetAlertRuleVersionFolders by grouping in query
**What is this feature?**
This PR implements a new Prometheus historian backend that allows Grafana alerting to write alert state history as Prometheus-compatible `ALERTS` metrics to remote Prometheus-compatible data sources.
The metric includes a few additional labels:
* `grafana_alertstate`: Grafana's full alert state, more granular than Prometheus.
* `grafana_rule_uid`: Grafana's alert rule UID.
Grafana states are included in the `grafana_alertstate` label also mapped to Prometheus-compatible `alertstate` values:
| Grafana alert state | `alertstate` | `grafana_alertstate` |
|---------------------|-----------------------|-----------------------|
| `Alerting` | `firing` | `alerting` |
| `Recovering` | `firing` | `recovering` |
| `Pending` | `pending` | `pending` |
| `Error` | `firing` | `error` |
| `NoData` | `firing` | `nodata` |
| `Normal` | _(no metric emitted)_ | _(no metric emitted)_ |
* refactor `selectorString` and remove Selector struct
* move code from selector string to BuildLogQuery
* batch requests by folder UID
* update historian annotation store to handle multiple queries
* sort folder uids to make consistent queries
* add logs to loki http
* log batch size but not content. content is logged by the client
* handle metadata map nil
* remove double context
* clean up logging in scheduler
* do not reuse loggers from previous ticks
* log the dropped tick
* log tick instead of ticknum
* replace with processing tick logs
* log sending notifications
* update logging in persister to fetch context
* logs to historian
moved them upstream to be able to log when store is overridden
* add method CanReadAllRules to rule authorization service
* add alias type Namespace for Folder in ngalert's models package. It implements the Namespacer interface that is used by authz logic
* update state history's backends to authorize access to rules.
* update Loki to add folders UIDs to query.
* Update BuildLogQuery to drop filter by folders if it's too long and fall back to in-memory filtering.
* Add TracedClient
* Handle errors and status codes
* Wire up tracing to normal ASH and loki annotation mapping
* Add tracing to remote alertmanager
* one more spot
* and not or
* More consistency with other grafana traces, lower cardinality name
* Implement keep last state for state transitions
* Respect For duration when keeping state
* Only keep transition from recording an annotation
* Add keep last state option for nodata/error in UI
* Move scope type vars to testutil package
* Expose parts of state historian for use in annotation backend
* Implement Loki ASH Annotation store
This store will only implement the `Get` method of a RepositoryImpl since alert state history
writes to Loki elsewhere.
* Use interface for Loki HTTP Client
* Add tests for Loki ASH Annotation store
* Add missing test
* Fix lint
* Organize tests
* Add filter tests
* Improve tests
* Move filter logic into outer function
* Fix lint
* Add comment
* Fix tests
* Fix lint
* Rename historian store + refactor
* Cleanup historian store
* Fix tests
* Minor cleanup
* Use new `ShouldRecordAnnotation` filter
* Fix logic and add tests for this check
* Fix typos, remove unused variables, `< 1` -> `== 0`
* More closely mimic RBAC filter from xorm to ensure correct logic
* Move off weaveworks client
* Address PR comments
* Alerting: Add metrics to the remote Alertmanager struct
* rephrase http_requests_failed description
* make linter happy
* remove unnecessary metrics
* extract timed client to separate package
* use histogram collector from dskit
* remove weaveworks dependency
* capture metrics for all requests to the remote Alertmanager (both clients)
* use the timed client in the MimirAuthRoundTripper
* HTTPRequestsDuration -> HTTPRequestDuration, clean up mimir client factory function
* refactor
* less git diff
* gauge for last readiness check in seconds
* initialize LastReadinesCheck to 0, tweak metric names and descriptions
* add counters for sync attempts/errors
* last config sync and last state sync timestamps (gauges)
* change latency metric name
* metric for remote Alertmanager mode
* code review comments
* move label constants to metrics package
* Exclude mapped nodata transitions when nodata mapped to OK
* Fix processEvalResults test
* Don't check NoDataState when filtering transition
* Add comment to explain purpose of separate function
---------
Co-authored-by: William Wernert <william.wernert@grafana.com>
* Add limit query parameter
* Drop copy paste comment
* Extend history query limit to 30 days and 250 entries
* Fix history log entries ordering
* Update no history message, add empty history test
---------
Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
* Add fresh context with timeout and same log properties, re-derive logger
* Unify timeout constants
* Move ctx after shortcut that got added through rebasing
* Unify timeouts
* Port opentracing's SpanFromContext and ContextFromSpan to the grafana tracing package
* Support both opentracing and otel variants
* Better document why we're creating a new ctx
* Add new func to FakeSpan which was added after rebase
* Support grafana-specific traceID key in both tracer implementations
Yuri Tseretyan
Matthew Jacobson
William WernertandYuri Tseretyan
Santiago
Vadim Stepanov
Alexander Akhmetov
Stephanie Hingtgen
Mariell Hoversholm
maicon
Steve Simpson
Gabriel MABILLE
Alexander Weaver
Ryan McKinley
Marcus Efraimsson
William Wernert