Commit Graph

60 Commits

Author SHA1 Message Date
Alexander Akhmetov bdd5bf0397 Alerting: Configurable queue and batch size in external alert manager notifier (#109831) 2025-08-19 09:32:10 +02:00
Serge Zaitsev a95fb3a37c Chore: Omit integration tests if short test flag is passed (#108777)
* omit integration tests if short test flag is passed

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/tests/api/alerting/api_ruler_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/cmd/grafana-cli/commands/datamigrations/to_unified_storage_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/services/ngalert/models/receivers_test.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* fix the rest

* false positive

---------

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-07-28 13:38:54 +02:00
Vadim Stepanov bccc980b90 Alerting: Notifiication history (#107644)
* Add unified_alerting.notification_history to ini files

* Parse notification history settings

* Move Loki client to a separate package

* Loki client: add params for metrics and traces

* add NotificationHistorian

* rm writeDuration

* remove RangeQuery stuff

* wip

* wip

* wip

* wip

* pass notification historian in tests

* unify loki settings

* unify loki settings

* add test

* update grafana/alerting

* make update-workspace

* add feature toggle

* fix configureNotificationHistorian

* Revert "add feature toggle"

This reverts commit de7af8f7

* add feature toggle

* more tests

* RuleUID

* fix metrics test

* met.Info.Set(0)
2025-07-17 14:26:26 +01:00
Serge Zaitsev f66a693438 Chore: Rename integration tests to follow the common convention (#105987)
* automatically rename integration tests to follow the common convention

* name tests differently

* alter column type to bigint

* update another column to bigint

* add another alter

* fix subquery for mysql
2025-06-29 16:56:24 +02:00
Fayzal Ghantiwala 67d28465de Alerting: Fix debug log line in External Alertmanager sender (#105855)
Fix log line
2025-05-22 21:07:12 +03:00
Santiago 180b67ca6c Remote Alertmanager: Make timeout configurable in alert senders (#105599) 2025-05-19 12:52:12 +02:00
beejeebus 8f79e4882f Replace usage of http.DefaultClient and http.DefaultTransport (#104135)
Remove usage of http.DefaultClient and http.DefaultTransport

Part of grafana/data-sources#484
2025-05-09 13:26:39 -04:00
Fayzal Ghantiwala 0c47d96195 Alerting: Add debug log for alert timestamps before sending to external Alertmanager (#105156)
Add debug log for alert with timestamps
2025-05-09 01:25:05 +03:00
Mariell Hoversholm 757be6365a CI: Bump golangci-lint to 2.0.2 (#103572) 2025-04-10 14:42:23 +02:00
Alexander Akhmetov 09a7f9ba1c Alerting: Update external alert manager notifier (#103530) 2025-04-07 17:09:23 +02:00
Steve Simpson 0ffb7518cd Alerting: Ignore external alert sending errors when shutting down. (#102705) 2025-03-25 13:50:23 +02:00
Santiago 39f212a965 Alerting: Call RLock() before reading sendAlertsTo map (#99812)
* Alerting: Call RLock() before reading sendAlertsTo map

* defer unlocking

* drive-tru fix for another lock

* less time holding the lock in SyncAndApplyConfigFromDatabase
2025-01-31 12:43:02 +01:00
ismail simsek 79fc26ea87 Chore: Remove prometheus replace directive (#98966)
* remove replace directive

* have an adapter

* go mod tidy

* make update-workspace

* make update-workspace
2025-01-15 14:04:14 +01:00
Santiago 9e408f842c Alerting: Skip sanitizing labels when sending alerts to the remote Alertmanager (#96251)
* Alerting: Skip sanitizing labels when sending alerts to the remote Alertmanager

* fix drone image name
2024-11-13 11:21:44 -03:00
Matthew Jacobson e86929eb0a Alerting: Managed receiver resource permission in config api (#93632)
* Alerting: Managed receiver resource permission in config api
2024-09-25 09:39:36 -04:00
Kristin Laemmert a117b090cf chore: preallocate slices where we have a good idea of requirements (#91596)
* chore: preallocate slices where we have a good idea of requirements

* pr feedback
2024-08-07 08:34:52 -04:00
Dave Henderson 6262c56132 chore(perf): Pre-allocate where possible (enable prealloc linter) (#88952)
* chore(perf): Pre-allocate where possible (enable prealloc linter)

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* fix TestAlertManagers_buildRedactedAMs

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* prealloc a slice that appeared after rebase

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-06-14 14:16:36 -04:00
Kyle Brandt a738cb42d8 Prometheus: Update dependency to v0.52.0 (#87809)
* Prometheus: Update dependency to v0.52.0

* go work sync

* fix panics in tests

* go work sync

* prometheus v0.52.0

* handle errors

* Update pkg/services/ngalert/sender/sender_test.go

Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>

* Update pkg/services/ngalert/sender/sender_test.go

Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>

---------

Co-authored-by: Santiago <santiagohernandez.1997@gmail.com>
Co-authored-by: ismail simsek <ismailsimsek09@gmail.com>
2024-05-28 15:22:20 +02:00
Steve Simpson ed42119907 Alerting: Pass metrics Registerer into NewExternalAlertmanagerSender. (#88313)
* Alerting: Pass metrics Registerer into NewExternalAlertmanagerSender.

I will work on a separate change to export the metrics from Grafana, this
is a little more complicated.

* Typo
2024-05-24 23:03:34 +02:00
Steve Simpson 8bcf589301 Alerting: Pass logger into NewExternalAlertmanagerSender (#88256) 2024-05-24 20:11:26 +02:00
Steve Simpson 8421919cb5 Alerting: Feature toggle to disallow sending alerts externally (#87982)
* Define feature toggle

* Implement feature toggle
2024-05-23 14:29:19 +02:00
Alexander Weaver 623ee3a2be Alerting: Only append /alertmanager when sending alerts to mimir targets if not already present (#85543)
Don't append alertmanager if not present
2024-04-04 11:58:41 -05:00
Dave Henderson 5687243d0b Feature Flags: use FeatureToggles interface where possible (#85131)
* Feature Flags: use FeatureToggles interface where possible

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* Replace TestFeatureToggles with existing WithFeatures

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
2024-04-04 12:22:31 -04:00
Matthew Jacobson 3ea5c08c88 Alerting: External AM fix parsing basic auth with escape characters (#84681) 2024-03-18 13:04:57 -04:00
Andres Martinez Gotor 265200799d Chore: Update grafana-plugin-sdk (#84289) 2024-03-12 17:13:23 +01:00
Yuri Tseretyan 1eebd2a4de Alerting: Support for simplified notification settings in rule API (#81011)
* Add notification settings to storage\domain and API models. Settings are a slice to workaround XORM mapping
* Support validation of notification settings when rules are updated

* Implement route generator for Alertmanager configuration. That fetches all notification settings.
* Update multi-tenant Alertmanager to run the generator before applying the configuration.

* Add notification settings labels to state calculation
* update the Multi-tenant Alertmanager to provide validation for notification settings

* update GET API so only admins can see auto-gen
2024-02-15 09:45:10 -05:00
Diego Augusto Molina 9c29e1a783 Alerting: Fix data races and improve testing (#81994)
* Alerting: fix race condition in (*ngalert/sender.ExternalAlertmanager).Run

* Chore: Fix data races when accessing members of *ngalert/state.FakeInstanceStore

* Chore: Fix data races in tests in ngalert/schedule and enable some parallel tests

* Chore: fix linters

* Chore: add TODO comment to remove loopvar once we move to Go 1.22
2024-02-14 12:45:39 -03:00
Ryan McKinley 0c6e409350 Chore: Update arrow and prometheus dependencies (#82215)
* update arrow and prometheus

* keep codeowner

* use compare

* use grafana-plugin-sdk-go v0.210.0

---------

Co-authored-by: ismail simsek <ismailsimsek09@gmail.com>
2024-02-13 01:50:25 +01:00
William Wernert 2203bc2a3d Alerting: Refactor provisioning tests/fakes (#81205)
* Fix up test Alertmanager config JSON

* Move fake AM config and provisioning stores to fakes package
2024-01-24 17:15:55 -05:00
Santiago 1f6575e65e Alerting: Test MOA in remote secondary mode (#79828) 2024-01-05 11:05:27 +01:00
Santiago 01add144b8 Alerting: Send alerts to the remote Alertmanager (#77034)
* Alerting: Rename remote.ExternalAlertmanager to remote.Alertmanager

* Alerting: Send alerts to the remote Alertmanager

* add ticker to readiness check, add tests

* use options when creating a new sender.ExternaAlertmanager

* unexport defaultMaxQueueCapacity

* delete unused defaultConfig field

* add debug log line when sending alerts to the remote alertmanager

* move and refactor readiness check

* update tests to not include defaultConfig
2023-10-25 11:52:48 +02:00
Santiago 61cb26711e Alerting: Fetch alerts from a remote Alertmanager (#75844)
* Alerting: post alerts to the remote Alertmanager and fetch them

* fix broken tests

* Alerting: Add Mimir Backend image to devenv (blocks)

* add alerting as code owner for mimir_backend block

* Alerting: Use Mimir image to run integration tests for the remote Alertmanager

* skip integration test when running all tests

* skipping integration test when no Alertmanager URL is provided

* fix bad host for mimir_backend

* remove basic auth testing until we have an nginx image in our CI

* add integration tests for alerts

* fix tests

* change SendCtx -> Send, add context.Context to Send, fix CI

* add reover() for functions from the Prometheus Alertmanager HTTP client that could panic

* add TODO to implement PutAlerts in a way that mimicks what Prometheus does

* fix log format
2023-10-19 11:27:37 +02:00
Matthew Jacobson 82f3127e23 Alerting: Move legacy alert migration from sqlstore migration to service (#72702) 2023-10-12 13:43:10 +01:00
Alexander Weaver f6649d7a97 Revert "Alerting: Remove vendored models in migration service" (#76387)
Revert "Alerting: Remove vendored models in migration service (#74503)"

This reverts commit 6a8649d544.
2023-10-11 14:21:21 -05:00
Matthew Jacobson 6a8649d544 Alerting: Remove vendored models in migration service (#74503)
This PR replaces the vendored models in the migration with their equivalent ngalert models. It also replaces the raw SQL selects and inserts with service calls.

It also fills in some gaps in the testing suite around:

    - Migration of alert rules: verifying that the actual data model (queries, conditions) are correct 9a7cfa9
    - Secure settings migration: verifying that secure fields remain encrypted for all available notifiers and certain fields migrate from plain text to encrypted secure settings correctly e7d3993

Replacing the checks for custom dashboard ACLs will be replaced in a separate targeted PR as it will be complex enough alone.
2023-10-11 17:22:09 +01:00
Ryan McKinley 025b2f3011 Chore: use any rather than interface{} (#74066) 2023-08-30 18:46:47 +03:00
guangwu bbe4b0d3de chore: remove refs to deprecated io/ioutil (#70300) 2023-06-22 12:19:23 +02:00
Emil Tullstedt 23a9963507 Chore: Upgrade Prometheus to 2.43.0 (#67853)
- github.com/prometheus/prometheus => 2.43.0 (aka 0.43.0)
- github.com/prometheus/client_golang => 1.15.1
2023-05-10 14:09:49 +02:00
Jean-Philippe Quéméner bbce69f295 Alerting: Use configured headers for external alertmanager (#63819) 2023-04-21 16:16:27 +02:00
Steve Simpson 04336d53a9 Alerting: Update prometheus version (#65688) 2023-03-31 16:34:35 +02:00
Yuri Tseretyan 98e1aeaebd Alerting: Fix client to external Alertmanager to correctly build URL for Mimir Alertmanager (#63676) 2023-02-23 13:55:26 -05:00
suntala 49b3027049 Chore: Remove Result field from datasources (#63048)
* Remove Result field from AddDataSourceCommand
* Remove DatasourcesPermissionFilterQuery Result
* Remove GetDataSourceQuery Result
* Remove GetDataSourcesByTypeQuery Result
* Remove GetDataSourcesQuery Result
* Remove GetDefaultDataSourceQuery Result
* Remove UpdateDataSourceCommand Result
2023-02-09 15:49:44 +01:00
Santiago ba731f7865 Alerting: Mark AM configuration as applied (#61330)
* Mark AM configuration as applied

* add missing checks, make linter happy

* fix deadlock, mark as valid on save and on load

* mark configurations only if needed

* check error after applyConfig()

* code review comments

* code review changes

* more code review changes

* clean HistoricConfigFromAlertConfig function
2023-02-02 14:45:17 -03:00
idafurjes 23c27cffb3 Chore: Rename Id to ID in alerting models (#62777)
* Chore: Rename Id to ID in alerting models

* Add xorm tags for datasource

* Add xorm tag for uid
2023-02-02 17:22:43 +01:00
Serge Zaitsev d6d4097567 Chore: Fix goimports grouping in alerting (#62424)
* fix goimports

* fix goimports order
2023-01-30 09:55:35 +01:00
Joe Blubaugh 1a8d0e2736 Alerting: Speed up unit and integration tests. (#60067)
This change marks tests in the `sender` package that use an external
process as integration tests instead of unit tests. This speeds up the
package's unit tests by about 20 seconds.

This change also reduces the number of alert instances in the `store`
package's bulk write integration test from 20_000 to 10_000. This is
still enough to exercise the bulk-write code but speeds up the package
tests from about 250s to 130s.

Put together, integration tests go to about 160s while also speeding up
unit tests by 20s.
2022-12-12 14:21:06 +08:00
Alex Moreno 45facbba11 Alerting: Remove url based external alertmanagers config (#57918)
* Remove URL-based alertmanagers from endpoint config

* WIP

* Add migration and alertmanagers from admin_configuration

* Empty comment removed

* set BasicAuth true when user is present in url

* Remove Alertmanagers from GET /admin_config payload

* Remove URL-based alertmanager configuration from UI

* Fix new uid generation in external alertmanagers migration

* Fix tests for URL-based external alertmanagers

* Fix API tests

* Add more tests, move migration code to separate file, and remove possible am duplicate urls

* Fix edge cases in migration

* Fix imports

* Remove useless fields and fix created_at/updated_at retrieval

Co-authored-by: George Robinson <george.robinson@grafana.com>
Co-authored-by: Konrad Lalik <konrad.lalik@grafana.com>
2022-11-10 16:34:13 +01:00
Alexander Weaver 5ee4744d62 Alerting: Improve operational logs in sender package (#57134)
* Audit logs in sender package

* Fix casing and touch up a few key names

* Avoid logging entire alert struct

* Log configuration ID being applied

* Revert change to errorf rather than log

* Tune levels further and remove some redundancies

* Adjust logger naming and standardize log context

* Adjust logger naming in router

* Move log and get rid of dead error handling code
2022-10-20 14:19:04 -05:00
Alexander Weaver 3ddb28bad9 Find-and-replace 'err' logs to 'error' to match log search conventions (#57309) 2022-10-19 17:36:54 -04:00
Matthew Jacobson 940d18ad57 Alerting: Sanitize invalid label/annotation names for external alertmanagers (#54537)
* Alerting: Sanitize invalid label/annotation names for external alertmanagers

Grafana's built-in Alertmanager supports both Unicode label keys and values; however, if using an external
Prometheus Alertmanager label keys must be compatible with their data model.
This means label keys must only contain ASCII letters, numbers, as well as underscores and match the regex
`[a-zA-Z_][a-zA-Z0-9_]*`.

Any invalid characters will now be removed or replaced by the Grafana alerting engine before being sent to
the external Alertmanager according to the following rules:

- `Whitespace` will be removed.
- `ASCII characters` will be replaced with `_`.
- `All other characters` will be replaced with their lower-case hex representation.

* Prefix hex replacements with `0x`

* Refactor for clarity

* Apply suggestions from code review

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>

Co-authored-by: brendamuir <100768211+brendamuir@users.noreply.github.com>
2022-09-07 11:39:39 -04:00