Commit Graph

783 Commits

Author SHA1 Message Date
owensmallwood f116539541 Unified Storage: Update readme (#114415)
* update readme

* Adds message about creating database

* update sample storage config
2025-11-26 15:53:00 -06:00
Matheus Macabu 21c1d9aedd Secrets: Remove unused methods and dependencies from secure value service (#114467) 2025-11-26 12:58:00 +01:00
Misi 93ec32dd6a IAM: Add teams/{id}/groups as a custom endpoint to Teams API (#114228)
* Add teams/{id}/groups as a custom endpoint

* TeamGroupsHandler OSS and Ent registration

* Update OpenAPI spec

* Add indexer tests for external group mapping

* Remove noopsearch

* Remove unnecessary interface declaration, fixes

* Chores

* fmt

* Rename constant

* Align the rest to the changes of main

* Update workspace

* Add missing file
2025-11-25 14:19:57 +01:00
Bruno 9091ac6f5c Secrets: add basic namespace and name checks to keeper store and secure value store (#114355) 2025-11-25 10:04:43 -03:00
Steve Simpson d50e2a94a3 Alerting: Implement setting of version message in convert API. (#114331)
Part 2 of adding version messages to the `alert_rule_version` able. This allows
setting the message via a header when using the Prometheus conversion API,
which can be useful for e.g. linking changes back to source control.
2025-11-25 11:20:28 +01:00
Jean-Philippe Quéméner b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Mustafa Sencer Özcan bca58f5626 feat: add folder tree validation for unified data migration (#114342)
* fix: add validator

* fix: revert startup index
2025-11-24 11:02:14 +01:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Matheus Macabu 5e949fc955 Secrets: Fix secure value creation timestamp changing when updating it (#114290) 2025-11-21 16:31:17 +01:00
Bruno 0d67442f1a Secrets manager: create secure value using the active keeper (#114039)
* Secrets manager: create secure value using the active keeper

* SecureValueService.Update: fetch secure value from db to get the keeper

* fix keeper_store_test.go

* SecureValueService: fix bug in update where the current version keeper wasn't being passed to the createNewVersion method

* make gofmt

* remove outdated test

* update TestModel

* undo enterprise_imports changes

* use xkube.Namespace

* migrator: set secret_secure_value.keeper to 'system' when the column is null

* indent cue

* fix tests

* fix enterprise imports

* properly fix enterprise imports

* make update-workspace

* go mod tidy

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-21 11:20:16 -03:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Mustafa Sencer Özcan af684248b5 feat: add reindex trigger as a migration validation (#114283)
* feat: add search validation

* chore: remove comment

* fix: remove local test

* fix: revert rebuild index portion

* chore: remove unnecessary comments

* refactor: validator

* fix: revert setting
2025-11-21 12:21:19 +01:00
Mustafa Sencer Özcan 30c04ab3fc feat: inject unified data migrations in dual writer (#114138)
* feat: draft changes for on-prem unified migration

* feat: further draft changes for on-prem unified migration

* fix: remove some tbis

* refactor: rename

* fix: another approach

* fix: background service related issues

* fix: address comments

* fix: make gen-go

* fix: background service related issues

* feat: refactor dual writer and legacy migrator

* fix: minor issues

* feat: working version in oss

* fix: wire

* fix: revert test data override

* fix: enterprise related issues

* chore: add todo

* fix: revert dual writer method

* fix: lint

* chore: logger format

* fix: reduce log level

* fix: log change

* fix: disable

* fix: address comments

* fix: return error on dual writer service

* fix: merge conflict

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2025-11-20 16:40:20 +01:00
Rafael Bortolon Paulovic 65fd15bbf9 feat: define unified disable migrations flag (#114120)
chore: add US data migrations config
2025-11-20 15:47:44 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
Mustafa Sencer Özcan 0d4ad01b65 feat: add unified data migrations for dashboard and folders (#113853) 2025-11-19 09:09:08 +01:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Rafael Bortolon Paulovic 0fc0fb8db5 chore: move enable search to US configuration (#114070) 2025-11-18 17:32:25 +01:00
Peter Štibraný ec74a721ea Be more tolerant to invalid JSON when parsing dashboards for indexing (#114040)
* Tags can only be string array.

* Be more lenient when parsing dashboard. Parse what we can, don't error out easily.

* Check element before parsing, log unexpected types.
2025-11-18 14:01:57 +01:00
Rafael Bortolon Paulovic 32da63a20f chore: remove US history pruner feature toggle (#114014) 2025-11-17 20:47:37 +01:00
Georges Chaudy 1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner e92a976de2 fix(unified-storage): process list items concurrently (#113801) 2025-11-13 12:08:14 +01:00
Mustafa Sencer Özcan 6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Mike 6c512dabdc Secrets: Fix MariaDB syntax error due to unsupported CTE syntax (#111610) (#113690)
* Secrets: fix MariaDB syntax error due to unsupported CTE syntax (#111610)

* parametrize guid/created columns and re-generate test fixtures

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-12 15:15:43 +01:00
Mustafa Sencer Özcan 735b776edc fix: cleanup legacy resource if it is created in legacy during dual update (#113753) 2025-11-12 15:02:36 +01:00
Jean-Philippe Quéméner b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner 76ab09a6a2 feat(unified-storage): add tracing to apistore (#113714) 2025-11-12 09:48:56 +01:00
Jean-Philippe Quéméner 589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Nathan Vērzemnieks f4b23253b1 DataSources: Update SDKs in support of auth service (#112101)
* DataSources: Update SDKs for auth service

* Fix deprecated methods & types for new arrow-go version
2025-11-07 10:15:27 +01:00
Will Assis c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Jean-Philippe Quéméner fd14d4a5ed feat(unified-storage): add tracing to dual writer and legacy storage (#113504)
Co-authored-by: Mustafa Sencer Özcan <32759850+mustafasencer@users.noreply.github.com>
2025-11-06 11:42:46 +00:00
Mustafa Sencer Özcan 51c31e00a4 fix: dual writer log object formatting (#113492)
fix: logging
2025-11-06 11:03:28 +01:00
Rafael Bortolon Paulovic 7281bb7069 fix: background delete on create failure after ctx cancellation (#113442)
* fix: background delete on create failure after ctx cancellation

* fix: address comments

* chore: remove tests using mock
2025-11-06 09:21:11 +01:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Georges Chaudy 07bf7b2ae1 kvstore: add cluster-scoped resource support (#113183)
kvstore add experimental clusterscope resource
2025-11-03 15:53:59 -08:00
Michael Mandrus cf242def3a Secrets: Refactor data_key_id out of the encoded secure value payload (#112077)
* Secrets: Refactor data_key_id out of the encoded secure value payload (#111852)

* everything compiles

* tests pass

* remove file included by accident

* add entry to gitignore

* some scaffolding for the migration executor

* remove file

* implement and test the migration

* use xkube.Namespace in our interfaces

* add todo

* update wire deps

* add some logs

* fix wire dependency ordering

* create tests to validate error conditions during migrations

* only run the migration as an MT api server

* formatting issues

* change detection of secrets running as MT server

* add todo

* use more specific initializer flags

* make secrets playwright tests work

* set new properties to true by default

* remove developer mode flag

* fix unit tests
2025-10-30 23:04:32 -04:00
Mustafa Sencer Özcan 58098f9339 chore: improve unified dual writer logging (#113203)
chore: improve dual writer logging
2025-10-30 10:12:29 +01:00
Georges Chaudy 9533cc4dbb kvstore: Fix missing Folder field in listModifiedSinceEventStore data lookup (#113131)
* fix: ListModifiedSince for resources in folders

* fix: ListModifiedSince for resources in folders
2025-10-28 23:18:15 +01:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy 147ff4279b kvstore: fix events lookback + startkey (#113092)
* fix snowflakes events

* add tests
2025-10-28 17:25:55 +01:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Ryan McKinley d4d8b2562e Chore: Update gocloud.dev, removing opencensus (#113056) 2025-10-28 09:10:37 +00:00
Georges Chaudy bc9540fadb kvstore: use batch delete to cleanup old events (#112737)
* use batchdelete for cleaning up old events

* comment
2025-10-27 12:27:31 +01:00
Denis Vodopianov 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Matheus Macabu 2c6c564c18 Secrets: Add decrypter to decrypt duration metric (#112942) 2025-10-24 15:23:54 +02:00
Peter Štibraný a4aa3529c8 Cleanup old entries from resource_last_import_time table. (#112438)
* Cleanup old entries from resource_last_import_time table.

* Add index for last_import_time column.

* Address review feedback.
2025-10-23 11:17:08 +00:00
Misi ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3391163f9c Check that we see less calls than allowed maximum. (#112785)
Check that we see less calls than allowed maximum.
2025-10-22 08:39:19 +00:00
Gábor Farkas 8ab7a20265 update dependency grafana/grafana-plugin-sdk-go to v0.281.0 (#112726)
* go get github.com/grafana/grafana-plugin-sdk-go@v0.281.0

* make update-workspace

* updated failing unit test

* disable deprecation warnings

* datasources: allow underscore-prefixed local time range

* updated go.work.sum
2025-10-22 09:02:11 +02:00