* Alerting: Fix contact point testing with secure settings
Fixes double encryption of secure settings during contact point testing and removes code duplication
that helped cause the drift between alertmanager and test endpoint. Also adds integration tests to cover
the regression.
Note: provisioningStore is created to remove cycle and the unnecessary dependency.
* Alerting: Make ApplyAlertmanagerConfiguration only decrypt/encrypt new/changed secure settings
Previously, ApplyAlertmanagerConfiguration would decrypt and re-encrypt all secure settings. However, this caused re-encrypted secure settings to be included in the raw configuration when applied to the embedded alertmanager, resulting in changes to the hash. Consequently, even if no actual modifications were made, saving any alertmanager configuration triggered an apply/restart and created a new historical entry in the database.
To address the issue, this modifies ApplyAlertmanagerConfiguration, which is called by POST `api/alertmanager/grafana/config/api/v1/alerts`, to decrypt and re-encrypt only new and updated secure settings. Unchanged secure settings are loaded directly from the database without alteration.
We determine whether secure settings have changed based on the following (already in-use) assumption: Only new or updated secure settings are provided via the POST `api/alertmanager/grafana/config/api/v1/alerts` request, while existing unchanged settings are omitted.
* Ensure saving a grafana-managed contact point will only send new/changed secure settings
Previously, when saving a grafana-managed contact point, empty string values were transmitted for all unset secure settings. This led to potential backend issues, as it assumed that only newly added or updated secure settings would be provided.
To address this, we now exclude empty ('', null, undefined) secure settings, unless there was a pre-existing entry in secureFields for that specific setting. In essence, this means we only transmit an empty secure setting if a previously configured value was cleared.
* Fix linting
* refactor omitEmptyUnlessExisting
* fixup
---------
Co-authored-by: Gilles De Mey <gilles.de.mey@gmail.com>
* Alerting: Add endpoint to revert to a previous alertmanager configuration
This endpoint is meant to be used in conjunction with /api/alertmanager/grafana/config/history to
revert to a previously applied alertmanager configuration. This is done by ID instead of raw config
string in order to avoid secure field complications.
* WIP
* skip invalid historic configurations instead of erroring
* add warning log when bad historic config is found
* remove unused custom marshaller for GettableHistoricUserConfig
* add id to historic user config, move limit check to store, fix typo
* swagger spec
* introduce alias for json.RawMessage with name RawMessage. This is needed to keep raw JSON and implement a marshaler for YAML, which does not seem to be used but there are tests that fail.
* replace usage of simplejson with RawMessage in NotificationChannelConfig
* remove usage of simplejson in tests
* change migration code to convert simplejson to raw message
* (WIP) switch to fork AM, first implementation of the API, generate spec
* get receivers avoiding race conditions
* use latest version of our forked AM, tests
* make linter happy, delete TODO comment
* update number of expected paths to += 2
* delete unused endpoint code, code review comments, tests
* Update pkg/services/ngalert/notifier/alertmanager.go
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
* remove call to fmt.Println
* clear naming for fields
* shorter variable names in GetReceivers
Co-authored-by: Matthew Jacobson <matthew.jacobson@grafana.com>
* generalize error handling in forking request handlers
* remove MatchesBackend and change test to test Can
* add 404 to route specs
* change backendTypeByUID to getDatasourceByUID of expected type
* use common errors in api testing
* handle 401 in errorToResponse
* replace backend type error with "unexpected datasource type"
* update swagger spec
* Alerting: Remove double quotes from matchers
With #38629 a new Alertmanager configuration object was introduced with `object_matchers`, it was meant to circumvent around the fact that Prometheus label names don't support a set of characters that Grafana needs to support for alerts, silences, matchers, etc. (with a common example being elasticsearch's `.`).
This new object does not include the label of sanitzation or validation that its Prometheus equivalent supports in `matchers` and therefore are semantically not equivalent.
This triggered the problem that when the migration is run, we use `matchers` as the object to populate in configuration for routing policies, but when the UI does its first save this object is transformed to `object_matchers`.
Matchers that were previously running just fine would immediately stop working as soon as the configuration is saved.
This problem surfaced with the introduction of #49952 where we stopped stripping double quotes from matchers (not just regex but _all_ of them).
* Add comment explaining rationale and future removal
Co-authored-by: Alex Weaver <weaver.alex.d@gmail.com>
* Add validator for mute timing and make it provisionable
* Add tests to ensure prometheus validators are running and errors are propagated
* Internal API for manipulating mute timings
* Define and generate API layer
* Wire up generated code
* Implement API handlers
* Tests for golang layer
* Fix reference bug
* Fix linter and auth tests
* Resolve semantic errors and regenerate
* Remove pointless comment
* Extract out provisioning path param keys, simplify
* Expected number of paths
* Refactor GET am config to be extensible
* Extract post config route
* Fix tests
* Remove temporary duplication
* Fix broken test due to layer shift
* Fix duplicated error message
* Properly return 400 on config rejection
* Revert weird half method extraction
* Move things to notifier package and avoid redundant interface
* Simplify documentation
* Split encryption service and depend on minimal abstractions
* Properly initialize things all the way up to the composition root
* Encryption -> Crypto
* Address misc feedback
* Missing docstring
* Few more simple polish improvements
* Unify on MultiOrgAlertmanager. Discover bug in existing test
* Fix rebase conflicts
* Misc feedback, renames, docs
* Access crypto hanging off MultiOrgAlertmanager rather than having a separate API to initialize
* Base-line API for provisioning notification policies
* Wire API up, some simple tests
* Return provenance status through API
* Fix missing call
* Transactions
* Clarity in package dependencies
* Unify receivers in definitions
* Fix issue introduced by receiver change
* Drop unused internal test implementation
* FGAC hooks for provisioning routes
* Polish, swap names
* Asserting on number of exposed routes
* Don't bubble up updated object
* Integrate with new concurrency token feature in store
* Back out duplicated changes
* Remove redundant tests
* Regenerate and create unit tests for API layer
* Integration tests for auth
* Address linter errors
* Put route behind toggle
* Use alternative store API and fix feature toggle in tests
* Fixes, polish
* Fix whitespace
* Re-kick drone
* Rename services to provisioning
* Alerting: Remove internal labels from prometheus compatible API responses
* Appease the linter
* Fix integration tests
* Fix API documentation & linter
* move removal of internal labels to the models
* Alerting: accept mute_timing_intervals through the api for the embedded alertmanager
* add workaround for mutetimeinterval
* add mute timings to routes
* revert changes
* Update pkg/services/ngalert/api/api_alertmanager.go
* Update pkg/services/ngalert/api/api_alertmanager.go
* Update pkg/services/ngalert/api/api_alertmanager.go
* update prometheus/alertmanager dependency
* add some var docs
* Use secrets service in pluginproxy
* Use secrets service in pluginxontext
* Use secrets service in pluginsettings
* Use secrets service in provisioning
* Use secrets service in authinfoservice
* Use secrets service in api
* Use secrets service in sqlstore
* Use secrets service in dashboardshapshots
* Use secrets service in tsdb
* Use secrets service in datasources
* Use secrets service in alerting
* Use secrets service in ngalert
* Break cyclic dependancy
* Refactor service
* Break cyclic dependancy
* Add FakeSecretsStore
* Setup Secrets Service in sqlstore
* Fix
* Continue secrets service refactoring
* Fix cyclic dependancy in sqlstore tests
* Fix secrets service references
* Fix linter errors
* Add fake secrets service for tests
* Refactor SetupTestSecretsService
* Update setting up secret service in tests
* Fix missing secrets service in multiorg_alertmanager_test
* Use fake db in tests and sort imports
* Use fake db in datasources tests
* Fix more tests
* Fix linter issues
* Attempt to fix plugin proxy tests
* Pass secrets service to getPluginProxiedRequest in pluginproxy tests
* Fix pluginproxy tests
* Revert using secrets service in alerting and provisioning
* Update decryptFn in alerting migration
* Rename defaultProvider to currentProvider
* Use fake secrets service in alert channels tests
* Refactor secrets service test helper
* Update setting up secrets service in tests
* Revert alerting changes in api
* Add comments
* Remove secrets service from background services
* Convert global encryption functions into vars
* Revert "Convert global encryption functions into vars"
This reverts commit 498eb19859.
* Add feature toggle for envelope encryption
* Rename toggle
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Co-authored-by: Joan López de la Franca Beltran <joanjan14@gmail.com>
* Encryption: Add support to encrypt/decrypt sjd
* Add datasources.Service as a proxy to datasources db operations
* Encrypt ds.SecureJsonData before calling SQLStore
* Move ds cache code into ds service
* Fix tlsmanager tests
* Fix pluginproxy tests
* Remove some securejsondata.GetEncryptedJsonData usages
* Add pluginsettings.Service as a proxy for plugin settings db operations
* Add AlertNotificationService as a proxy for alert notification db operations
* Remove some securejsondata.GetEncryptedJsonData usages
* Remove more securejsondata.GetEncryptedJsonData usages
* Fix lint errors
* Minor fixes
* Remove encryption global functions usages from ngalert
* Fix lint errors
* Minor fixes
* Minor fixes
* Remove securejsondata.DecryptedValue usage
* Refactor the refactor
* Remove securejsondata.DecryptedValue usage
* Move securejsondata to migrations package
* Move securejsondata to migrations package
* Minor fix
* Fix integration test
* Fix integration tests
* Undo undesired changes
* Fix tests
* Add context.Context into encryption methods
* Fix tests
* Fix tests
* Fix tests
* Trigger CI
* Fix test
* Add names to params of encryption service interface
* Remove bus from CacheServiceImpl
* Add logging
* Add keys to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Add missing key to logger
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
* Undo changes in markdown files
* Fix formatting
* Add context to secrets service
* Rename decryptSecureJsonData to decryptSecureJsonDataFn
* Name args in GetDecryptedValueFn
* Add template back to NewAlertmanagerNotifier
* Copy GetDecryptedValueFn to ngalert
* Add logging to pluginsettings
* Fix pluginsettings test
Co-authored-by: Tania B <yalyna.ts@gmail.com>
Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
Remove validation for labels to be accepted in the Alertmanager, This helps with datasources that produce non-compatible labels.
Adds an "object_matchers" to alert manager routers so we can support labels names with extended characters beyond prometheus/openmetrics. It only does this for the internal Grafana managed Alert Manager.
This requires a change to alert manager, so for now we use grafana/alertmanager which is a slight fork, with the intention of going back to upstream.
The frontend handles the migration of "matchers" -> "object_matchers" when the route is edited and saved. Once this is done, downgrades will not work old versions will not recognize the "object_matchers".
Co-authored-by: Kyle Brandt <kyle@grafana.com>
Co-authored-by: Nathan Rodman <nathanrodman@gmail.com>
This commit adds contact point testing to ngalerts via a new API
endpoint. This endpoint accepts JSON containing a list of
receiver configurations which are validated and then tested
with a notification for a test alert. The endpoint returns JSON
for each receiver with a status and error message. It accepts
a configurable timeout via the Request-Timeout header (in seconds)
up to a maximum of 30 seconds.
* Alerting: Implement /status for the notification system
Implements the necessary plumbing to have a /status endpoint on the
notification system.
* Add API examples
* Update API specs
* Update prometheus/common dependency
Co-authored-by: Sofia Papagiannaki <sofia@grafana.com>
