* Alerting: Update grafana/alerting
* make tests pass by implementing yaml unmarshallers and deleting fields with omitempty in their yaml tags
* go mod tidy
* fix tests by implementing not calling GettableApiAlertingConfig.UnmarshalYAML from GettableApiAlertingConfig.UnmarshalJSON
* cleanup, reduce diff
* fix more tests
* update grafana/alerting to latest commit, delete global section from configs in tests
* bring back YAML unmarshaller for GettableApiAlertingConfig
* update alerting package dependency to point to main
* skip test for sns notifier
* Alerting: Implement GetStatus in the remote Alertmanager struct
* update tests
* fix tests, extract AlertmanagerConfig from PostableConfig
* get the remote AM config instead of the Grafana one from the remote AM
* pass grafana AM config in test
* return error in GetStatus instead of logging it (internal AM)
* Alerting: Persist silence state immediately on Create/Delete
Persists the silence state to the kvstore immediately instead of waiting for the
next maintenance run. This is used after Create/Delete to prevent silences from
being lost when a new Alertmanager is started before the state has persisted.
This can happen, for example, in a rolling deployment scenario.
* Fix test that requires real data
* Don't error if silence state persist fails, maintenance will correct
* Add notification settings to storage\domain and API models. Settings are a slice to workaround XORM mapping
* Support validation of notification settings when rules are updated
* Implement route generator for Alertmanager configuration. That fetches all notification settings.
* Update multi-tenant Alertmanager to run the generator before applying the configuration.
* Add notification settings labels to state calculation
* update the Multi-tenant Alertmanager to provide validation for notification settings
* update GET API so only admins can see auto-gen
* Alerting: Introduce a Mimir client as part of the Remote Alertmanager
Mimir client that understands the new APIs developed for mimir. Very much a WIP still.
* more wip
* appease the linter
* more linting
* add more code
* get state from kvstore, encode, send
* send state to the remote Alertmanager, extract fullstate logic into its own function
* pass kvstore to remote.NewAlertmanager()
* refactor
* add fake kvstore to tests
* tests
* use FileStore to get state
* always log 'completed state upload'
* refactor compareRemoteConfig
* base64-encode the state in the file store
* export silences and nflog filenames, refactor
* log 'completed state/config upload...' regardless of outcome
* add values to the state store in tests
* address code review comments
* log error from filestore
---------
Co-authored-by: gotjosh <josue.abreu@gmail.com>
This PR replaces the vendored models in the migration with their equivalent ngalert models. It also replaces the raw SQL selects and inserts with service calls.
It also fills in some gaps in the testing suite around:
- Migration of alert rules: verifying that the actual data model (queries, conditions) are correct 9a7cfa9
- Secure settings migration: verifying that secure fields remain encrypted for all available notifiers and certain fields migrate from plain text to encrypted secure settings correctly e7d3993
Replacing the checks for custom dashboard ACLs will be replaced in a separate targeted PR as it will be complex enough alone.
* Alerting: Add endpoint to revert to a previous alertmanager configuration
This endpoint is meant to be used in conjunction with /api/alertmanager/grafana/config/history to
revert to a previously applied alertmanager configuration. This is done by ID instead of raw config
string in order to avoid secure field complications.
* WIP
* skip invalid historic configurations instead of erroring
* add warning log when bad historic config is found
* remove unused custom marshaller for GettableHistoricUserConfig
* add id to historic user config, move limit check to store, fix typo
* swagger spec
* Mark AM configuration as applied
* add missing checks, make linter happy
* fix deadlock, mark as valid on save and on load
* mark configurations only if needed
* check error after applyConfig()
* code review comments
* code review changes
* more code review changes
* clean HistoricConfigFromAlertConfig function
* Test composition simplification from last PR
* Policies use proper API model everywhere
* Expose policy provenance in API, miss some dep injection
* Complete injection
* fix args
* Tests for provenance value
* Extract test helpers so tests are very readable
* Single source adapter struct that was copied in 3 places
* Drop redundant test
* Resolve merge conflicts on changelog
* Refactor GET am config to be extensible
* Extract post config route
* Fix tests
* Remove temporary duplication
* Fix broken test due to layer shift
* Fix duplicated error message
* Properly return 400 on config rejection
* Revert weird half method extraction
* Move things to notifier package and avoid redundant interface
* Simplify documentation
* Split encryption service and depend on minimal abstractions
* Properly initialize things all the way up to the composition root
* Encryption -> Crypto
* Address misc feedback
* Missing docstring
* Few more simple polish improvements
* Unify on MultiOrgAlertmanager. Discover bug in existing test
* Fix rebase conflicts
* Misc feedback, renames, docs
* Access crypto hanging off MultiOrgAlertmanager rather than having a separate API to initialize
* pass notification service down to the notifiers
* add ns to all notifiers
* remove bus from ngalert notifiers
* use smaller interfaces for notificationservice
* attempt to fix the tests
* remove unused struct field
* simplify notification service mock
* trying to resolve issues in the tests
* make linter happy
* make linter even happier
* linter, you are annoying
* Create API test for overwriting invalid alertmanager config
* Avoid requiring alertmanager readiness for config changes
* AlertmanagerSrv depends on functionality rather than concrete types
* Add test for non-ready alertmanagers
* Additional cleanup and polish
* Back out previous integration test changes
* Refactor of tests incorrectly caused a test to become redundant
* Use pre-existing fake secret service
* Drop unused interface
* Test against concrete MultiOrgAlertmanager re-using fake infra from other tests
* Fix linter error
* Empty commit to rerun checks
* Use secrets service in pluginproxy
* Use secrets service in pluginxontext
* Use secrets service in pluginsettings
* Use secrets service in provisioning
* Use secrets service in authinfoservice
* Use secrets service in api
* Use secrets service in sqlstore
* Use secrets service in dashboardshapshots
* Use secrets service in tsdb
* Use secrets service in datasources
* Use secrets service in alerting
* Use secrets service in ngalert
* Break cyclic dependancy
* Refactor service
* Break cyclic dependancy
* Add FakeSecretsStore
* Setup Secrets Service in sqlstore
* Fix
* Continue secrets service refactoring
* Fix cyclic dependancy in sqlstore tests
* Fix secrets service references
* Fix linter errors
* Add fake secrets service for tests
* Refactor SetupTestSecretsService
* Update setting up secret service in tests
* Fix missing secrets service in multiorg_alertmanager_test
* Use fake db in tests and sort imports
* Use fake db in datasources tests
* Fix more tests
* Fix linter issues
* Attempt to fix plugin proxy tests
* Pass secrets service to getPluginProxiedRequest in pluginproxy tests
* Fix pluginproxy tests
* Revert using secrets service in alerting and provisioning
* Update decryptFn in alerting migration
* Rename defaultProvider to currentProvider
* Use fake secrets service in alert channels tests
* Refactor secrets service test helper
* Update setting up secrets service in tests
* Revert alerting changes in api
* Add comments
* Remove secrets service from background services
* Convert global encryption functions into vars
* Revert "Convert global encryption functions into vars"
This reverts commit 498eb19859.
* Add feature toggle for envelope encryption
* Rename toggle
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>
* Alerting: Fixes a bug when trying to sync broken alertmanager config
Broken alertmanager configuration has the potential to be introduced as part of a migration e.g. due to incompatible data between what grafana accepts and what the Alertmanager expects. When this happens, we expect an eventually consistent behaviour where we'll keep trying to apply the configuration until it works.
As part of change in https://github.com/grafana/grafana/pull/39237 we introduced a regression that modified this behaviour and instead tried to create a new Alertmanager for that organization everytime, which eventually ended up in a panic due to a duplicate metrics being registered.
This PR fixes that and introduces a test to catch further regressions.
* Remove disable orgs
* Encryption: Add support to encrypt/decrypt sjd
* Add datasources.Service as a proxy to datasources db operations
* Encrypt ds.SecureJsonData before calling SQLStore
* Move ds cache code into ds service
* Fix tlsmanager tests
* Fix pluginproxy tests
* Remove some securejsondata.GetEncryptedJsonData usages
* Add pluginsettings.Service as a proxy for plugin settings db operations
* Add AlertNotificationService as a proxy for alert notification db operations
* Remove some securejsondata.GetEncryptedJsonData usages
* Remove more securejsondata.GetEncryptedJsonData usages
* Fix lint errors
* Minor fixes
* Remove encryption global functions usages from ngalert
* Fix lint errors
* Minor fixes
* Minor fixes
* Remove securejsondata.DecryptedValue usage
* Refactor the refactor
* Remove securejsondata.DecryptedValue usage
* Move securejsondata to migrations package
* Move securejsondata to migrations package
* Minor fix
* Fix integration test
* Fix integration tests
* Undo undesired changes
* Fix tests
* Add context.Context into encryption methods
* Fix tests
* Fix tests
* Fix tests
* Trigger CI
* Fix test
* Add names to params of encryption service interface
* Remove bus from CacheServiceImpl
* Add logging
* Add keys to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Add missing key to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Undo changes in markdown files
* Fix formatting
* Add context to secrets service
* Rename decryptSecureJsonData to decryptSecureJsonDataFn
* Name args in GetDecryptedValueFn
* Add template back to NewAlertmanagerNotifier
* Copy GetDecryptedValueFn to ngalert
* Add logging to pluginsettings
* Fix pluginsettings test
Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Alerting: Refactor & fix unified alerting metrics structure
Fixes and refactors the metrics structure we have for the ngalert service. Now, each component has its own metric struct that includes the JUST the metrics it uses. Additionally, I have fixed the configuration metrics and added new metrics to determine if we have discovered and started all the necessary configurations of an instance.
This allows us to alert on `grafana_alerting_discovered_configurations - grafana_alerting_active_configurations != 0` to know whether an alertmanager instance did not start successfully.
* Alerting: Persist notification log and silences to the database
This removes the dependency of having persistent disk to run grafana alerting. Instead of regularly flushing the notification log and silences to disk we now flush the binary content of those files to the database encoded as a base64 string.
Introduces org-level isolation for the Alertmanager and its components.
Silences, Alerts and Contact points are not separated by org and are not shared between them.
Co-authored with @davidmparrott and @papagian
Vadim Stepanov
Yuri Tseretyan
Alexander Akhmetov
Matthew Jacobson
Santiago
Dave Henderson
William Wernert
Alexander Weaver
Serge Zaitsev
Eng Zer Jun
Jean-Philippe Quéméner
gotjosh
Marcus Efraimsson
David Parrott