Commit Graph

815 Commits

Author SHA1 Message Date
Ryan McKinley d02b2a35cd Provisioning: Ignore dashboard change warning after save (#115401) 2025-12-17 10:17:57 +00:00
Rafael Bortolon Paulovic aa3b9dc4da Unified: Run resource data migrations at startup (#114857)
* chore: uncomment unified migration

* chore: adapt and fix tests

* chore: dynamically bump max conns if needed during migration

* chore: copilot suggestions

* chore: pass ctx in RegisterMigration

* chore: make playlists opt-out and dashboards opt-in

* chore: adjust dashboard test

* chore: disable enable log in test

* chore: address review comments

- do not use pointer config
- add migration registry

* chore: more consistent naming

* chore: fix playlist discovery test
2025-12-17 10:09:57 +01:00
Renato Costa c647140475 unified-storage: add Delete support to the sqlkv implementation (#115443) 2025-12-16 16:19:45 -05:00
Renato Costa bd8f0083ce unified-storage: add Get support to the sqlkv implementation (#115382) 2025-12-16 14:32:13 -05:00
Mustafa Sencer Özcan 2d6c1c4e9e docs: add readme for unified storage on-prem migrations (#114397)
* docs: add documentation for unified storage migrations

* docs: move

* docs: rename title

* docs: add docs

* fix: update table

* fix: lint

* docs: add migration table explanation
2025-12-16 08:00:22 +00:00
Will Assis 12dd3dffe0 unified-storage: sqlkv skeleton (#115176)
* implement sqlkv skeleton and include sqlkv in badgerkv tests
2025-12-15 08:56:15 -05:00
Will Assis 1b59c82b74 Revert "Unified-storage: sql backend key path backfill (#115033)" (#115257)
This reverts commit b2dd095bd8.
2025-12-12 17:00:08 +00:00
Will Assis b2dd095bd8 Unified-storage: sql backend key path backfill (#115033)
* unified-storage: add migration to backfill key_path in resource_history
2025-12-12 08:09:51 -05:00
Misi 0c264b7a5f IAM: Add user search endpoint (#114542)
* wip: initial changes, api registration

* wip

* LegacySearch working with sorting

* Revert mapper change for now

* Clean up

* Cleanup, add integration tests

* Improve tests

* OpenAPI def regen

* Use wildcard search, fix lastSeenAt handling, add lastSeenAtAge

* Add missing files

* Fix merge

* Fixes

* Add tests, regen openapi def

* Address feedback

* Address feedback batch 2

* Chores

* regen openapidef

* Address feedback

* Add tests for paging

* gen apis

* Revert go.mod, go.sum. go.work.sum

* Fix + remove extra tracer parameter
2025-12-11 19:54:48 +01:00
Ryan McKinley a6043deb33 UnifiedStorage: Include RV when fieldSelectors are processed in the backend (#115110) 2025-12-11 09:15:01 +03:00
Will Assis 30045c02c0 unified-storage: add index on resource_history key_path column (#115113) 2025-12-10 16:03:58 -05:00
owensmallwood 5e7b900416 Unified Storage: Adds readme for setting up quotas/overrides (#115031)
* adds readme for setting up quotas/overrides

* updates namespace wording

* updates docs

* update test

* Revert "update test"

This reverts commit ad43e355ba.
2025-12-10 09:21:52 -06:00
Will Assis 755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwood 6b50e2d730 Unified Storage: Update yaml decoding for quotas to accomodate top-level overrides key (#115049)
* update yaml decoding for quotas to accomodate top-level overrides key

* update test

* fix test indentation
2025-12-09 23:14:22 +00:00
owensmallwood b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwood a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Alexander Akhmetov c59d5d1c8e Alerting: Store instance annotations in alert rule state (#114975)
Alerting: Store annotations in alert instance state
2025-12-09 13:52:42 +01:00
Jean-Philippe Quéméner 5b9965ee47 fix(unified-storage): check auth before getting value (#114995) 2025-12-09 12:20:37 +01:00
Mihai Doarna b2e1b257b3 IAM: Add search for teams in app platform (#113503)
* add legacy search (wip)

* fix search field name

* implement team search endpoint

* generate openapi spec

* generate endpoints for frontend

* minor fixes

* fix issues found while testing

* add more fields to search result

* add basic unit tests

* add more unit tests

* improve getColumns() func in legacy search

* configure search endpoint in team.cue

* add team search handler

* add the searchTeams endpoint to manifest.cue

* make gofmt

* update openapi spec

* generate frontend endpoints

* remove unused field

* move fields defiitions to separate builder

* fix legacy search

* fix unit tests

* fix unit test

* address feedback

* fix unit test

* update openapi specs

* yarn generate-apis

* add missing unit tests
2025-12-08 15:02:59 +02:00
Ryan McKinley 78b1ae4f27 Search: Fix field selector parsing (#114940) 2025-12-06 16:45:18 +03:00
Rafael Bortolon Paulovic ff97bfc772 fix(unified): key_path column default (#114859)
fix: key_path column default
2025-12-04 12:40:21 -05:00
Mustafa Sencer Özcan 4c5d9cb95f feat: add unified storage data migration step for playlists (#114582)
* fix: add type

* feat: register step

* feat: add playlist support

* test: add test case

* fix: gen mock

* fix: go gen

* fix: lint

* fix: lint

* fix: tests

* fix: add resource

* fix: readd

* fix: address comments

* fix: independent playlist query for migrations

* fix: remove lock logic for sqlite

* fix: handle creation and update datetimes

* fix: query templating

* fix: simply resources and address comments
2025-12-04 15:15:00 +00:00
Will Assis ae4d2324d6 Unified storage kvstore bulk import support (#113791)
* implement batchdelete in datastore

* implement bulkprocess in kv storage_backend

* convert bulkRVs to snowflake
2025-12-04 09:26:50 -05:00
Will Assis 994e1dd58f unified-storage: sqlkv migrations (#114790)
* unified-storage: create resource_events table and add key_path column to resource_history

* Update resource_history_insert template

* update test snapshots

* use latin encoding for key_path and bump size to 2048
2025-12-04 09:21:17 -05:00
Renato Costa 3cb03b888f fix: apply appropriate labels and annotations when creating provisioned folders (#114669)
fix: labels and anotations when creating provisioned folders
2025-12-03 09:18:18 -05:00
Will Assis 047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
Rafael Bortolon Paulovic 31eaf1e898 chore: add log and metric before unified migration enforcement (#114598) 2025-12-01 17:56:59 +01:00
Rafael Bortolon Paulovic 12c6d7e83f fix(unified): in-proc SQLite data migration (#114537)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface

* fix: unified SQLite migration with SQLStore migrator

* revert changes to newResourceDBProvider
2025-11-28 13:13:35 +01:00
Georges Chaudy 5626dc50f8 feat(unified-storage): Add adaptive backoff to event notifier polling (#114401)
* use exponential backoff in notifier

* Enhance BadgerDB configuration in REST options with memory table size and number of memtables

* Enhance BadgerDB configuration in REST options by adding value threshold for LSM vs value log storage
2025-11-27 18:42:01 +01:00
owensmallwood df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
Rafael Bortolon Paulovic 4c869a21a4 feat(unified): data migration integration tests (#114418)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface
2025-11-27 13:35:49 +01:00
owensmallwood f116539541 Unified Storage: Update readme (#114415)
* update readme

* Adds message about creating database

* update sample storage config
2025-11-26 15:53:00 -06:00
Matheus Macabu 21c1d9aedd Secrets: Remove unused methods and dependencies from secure value service (#114467) 2025-11-26 12:58:00 +01:00
Misi 93ec32dd6a IAM: Add teams/{id}/groups as a custom endpoint to Teams API (#114228)
* Add teams/{id}/groups as a custom endpoint

* TeamGroupsHandler OSS and Ent registration

* Update OpenAPI spec

* Add indexer tests for external group mapping

* Remove noopsearch

* Remove unnecessary interface declaration, fixes

* Chores

* fmt

* Rename constant

* Align the rest to the changes of main

* Update workspace

* Add missing file
2025-11-25 14:19:57 +01:00
Bruno 9091ac6f5c Secrets: add basic namespace and name checks to keeper store and secure value store (#114355) 2025-11-25 10:04:43 -03:00
Steve Simpson d50e2a94a3 Alerting: Implement setting of version message in convert API. (#114331)
Part 2 of adding version messages to the `alert_rule_version` able. This allows
setting the message via a header when using the Prometheus conversion API,
which can be useful for e.g. linking changes back to source control.
2025-11-25 11:20:28 +01:00
Jean-Philippe Quéméner b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Mustafa Sencer Özcan bca58f5626 feat: add folder tree validation for unified data migration (#114342)
* fix: add validator

* fix: revert startup index
2025-11-24 11:02:14 +01:00
João Calisto c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Matheus Macabu 5e949fc955 Secrets: Fix secure value creation timestamp changing when updating it (#114290) 2025-11-21 16:31:17 +01:00
Bruno 0d67442f1a Secrets manager: create secure value using the active keeper (#114039)
* Secrets manager: create secure value using the active keeper

* SecureValueService.Update: fetch secure value from db to get the keeper

* fix keeper_store_test.go

* SecureValueService: fix bug in update where the current version keeper wasn't being passed to the createNewVersion method

* make gofmt

* remove outdated test

* update TestModel

* undo enterprise_imports changes

* use xkube.Namespace

* migrator: set secret_secure_value.keeper to 'system' when the column is null

* indent cue

* fix tests

* fix enterprise imports

* properly fix enterprise imports

* make update-workspace

* go mod tidy

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-21 11:20:16 -03:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Mustafa Sencer Özcan af684248b5 feat: add reindex trigger as a migration validation (#114283)
* feat: add search validation

* chore: remove comment

* fix: remove local test

* fix: revert rebuild index portion

* chore: remove unnecessary comments

* refactor: validator

* fix: revert setting
2025-11-21 12:21:19 +01:00
Mustafa Sencer Özcan 30c04ab3fc feat: inject unified data migrations in dual writer (#114138)
* feat: draft changes for on-prem unified migration

* feat: further draft changes for on-prem unified migration

* fix: remove some tbis

* refactor: rename

* fix: another approach

* fix: background service related issues

* fix: address comments

* fix: make gen-go

* fix: background service related issues

* feat: refactor dual writer and legacy migrator

* fix: minor issues

* feat: working version in oss

* fix: wire

* fix: revert test data override

* fix: enterprise related issues

* chore: add todo

* fix: revert dual writer method

* fix: lint

* chore: logger format

* fix: reduce log level

* fix: log change

* fix: disable

* fix: address comments

* fix: return error on dual writer service

* fix: merge conflict

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2025-11-20 16:40:20 +01:00
Rafael Bortolon Paulovic 65fd15bbf9 feat: define unified disable migrations flag (#114120)
chore: add US data migrations config
2025-11-20 15:47:44 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
Mustafa Sencer Özcan 0d4ad01b65 feat: add unified data migrations for dashboard and folders (#113853) 2025-11-19 09:09:08 +01:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Rafael Bortolon Paulovic 0fc0fb8db5 chore: move enable search to US configuration (#114070) 2025-11-18 17:32:25 +01:00