Commit Graph

804 Commits

Author SHA1 Message Date
owensmallwood
5e7b900416 Unified Storage: Adds readme for setting up quotas/overrides (#115031)
* adds readme for setting up quotas/overrides

* updates namespace wording

* updates docs

* update test

* Revert "update test"

This reverts commit ad43e355ba.
2025-12-10 09:21:52 -06:00
Will Assis
755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
owensmallwood
6b50e2d730 Unified Storage: Update yaml decoding for quotas to accomodate top-level overrides key (#115049)
* update yaml decoding for quotas to accomodate top-level overrides key

* update test

* fix test indentation
2025-12-09 23:14:22 +00:00
owensmallwood
b8acfade21 Unified Storage: Adds debug logs for checking quotas (#115036)
* adds debug logs for checking quotas

* make update-workspace
2025-12-09 18:59:46 +00:00
owensmallwood
a3daf0e39d Unified storage: Add quotas app to apiserver (#114425)
* initial generation

* went through doc to add new resource

* added dummy kind so grafana will run

* added dummy handler and custom route

* fix app name

* gets custom route working - still a dummy route

* adds groupOverride to manifest

* adds quotas to grpc client and server

* WIP - trying to get api recognized - not working

* Gets route working

* fixes group and resource vars

* expects group and resource as separate params

* set content-type header on response

* removes Quotas kind and regens

* Update grafana-app-sdk to v0.48.5

* Update codegen

* updates manifest

* formatting

* updates grafana-app-sdk version to 0.48.5

* regen ResourceClient mocks

* adds tests

* remove commented code

* uncomment go mod tidy

* fix tests and make update workspace

* adds quotas app to codeowners

* formatting

* make gen-apps

* deletes temp file

* fix generated folder code

* make gofmt

* make gen-go

* make update-workspace

* add COPY apps/quotas to Dockerfile

* fix test mock

* fixes undefined NewFolderStatus()

* make gen-apps, and add func for NewFolderStatus

* make gen-apps again

* make update-workspace

* regen folder_object_gen.go

* gofmt

* fix linting

* apps/folder make update-workspace

* make gen-apps

* make gen-apps

* fixes enterprise_imports.go

* go get testcontainers

* adds feature toggle

* make update-workspace

* fix go mod

* fix another client mock

---------

Co-authored-by: Steve Simpson <steve@grafana.com>
2025-12-09 09:40:34 -06:00
Jean-Philippe Quéméner
1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Alexander Akhmetov
c59d5d1c8e Alerting: Store instance annotations in alert rule state (#114975)
Alerting: Store annotations in alert instance state
2025-12-09 13:52:42 +01:00
Jean-Philippe Quéméner
5b9965ee47 fix(unified-storage): check auth before getting value (#114995) 2025-12-09 12:20:37 +01:00
Mihai Doarna
b2e1b257b3 IAM: Add search for teams in app platform (#113503)
* add legacy search (wip)

* fix search field name

* implement team search endpoint

* generate openapi spec

* generate endpoints for frontend

* minor fixes

* fix issues found while testing

* add more fields to search result

* add basic unit tests

* add more unit tests

* improve getColumns() func in legacy search

* configure search endpoint in team.cue

* add team search handler

* add the searchTeams endpoint to manifest.cue

* make gofmt

* update openapi spec

* generate frontend endpoints

* remove unused field

* move fields defiitions to separate builder

* fix legacy search

* fix unit tests

* fix unit test

* address feedback

* fix unit test

* update openapi specs

* yarn generate-apis

* add missing unit tests
2025-12-08 15:02:59 +02:00
Ryan McKinley
78b1ae4f27 Search: Fix field selector parsing (#114940) 2025-12-06 16:45:18 +03:00
Rafael Bortolon Paulovic
ff97bfc772 fix(unified): key_path column default (#114859)
fix: key_path column default
2025-12-04 12:40:21 -05:00
Mustafa Sencer Özcan
4c5d9cb95f feat: add unified storage data migration step for playlists (#114582)
* fix: add type

* feat: register step

* feat: add playlist support

* test: add test case

* fix: gen mock

* fix: go gen

* fix: lint

* fix: lint

* fix: tests

* fix: add resource

* fix: readd

* fix: address comments

* fix: independent playlist query for migrations

* fix: remove lock logic for sqlite

* fix: handle creation and update datetimes

* fix: query templating

* fix: simply resources and address comments
2025-12-04 15:15:00 +00:00
Will Assis
ae4d2324d6 Unified storage kvstore bulk import support (#113791)
* implement batchdelete in datastore

* implement bulkprocess in kv storage_backend

* convert bulkRVs to snowflake
2025-12-04 09:26:50 -05:00
Will Assis
994e1dd58f unified-storage: sqlkv migrations (#114790)
* unified-storage: create resource_events table and add key_path column to resource_history

* Update resource_history_insert template

* update test snapshots

* use latin encoding for key_path and bump size to 2048
2025-12-04 09:21:17 -05:00
Renato Costa
3cb03b888f fix: apply appropriate labels and annotations when creating provisioned folders (#114669)
fix: labels and anotations when creating provisioned folders
2025-12-03 09:18:18 -05:00
Will Assis
047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
Rafael Bortolon Paulovic
31eaf1e898 chore: add log and metric before unified migration enforcement (#114598) 2025-12-01 17:56:59 +01:00
Rafael Bortolon Paulovic
12c6d7e83f fix(unified): in-proc SQLite data migration (#114537)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface

* fix: unified SQLite migration with SQLStore migrator

* revert changes to newResourceDBProvider
2025-11-28 13:13:35 +01:00
Georges Chaudy
5626dc50f8 feat(unified-storage): Add adaptive backoff to event notifier polling (#114401)
* use exponential backoff in notifier

* Enhance BadgerDB configuration in REST options with memory table size and number of memtables

* Enhance BadgerDB configuration in REST options by adding value threshold for LSM vs value log storage
2025-11-27 18:42:01 +01:00
owensmallwood
df2f528612 Unified Storage: Adds overrides service to resource server (#113794)
* first pass of adding quotas service resource server

* passes prom reg as param

init quota service as part of server params

* init quota service as part of server params

* adds config and only creates quota service when overrides file path is defined

* when quota service enabled, check quota on create and log result

* update log message

* adds tests for quota service

* adds tests for config reloading when the file changes

* fix linter errors

* fix comment

* use startAndAwaitRunning

* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.

* adds tracing to quotas service

* adds nsr attributes to traces when getting quotas and resource stats

* update comment

* update comment

remove check for nil overrides since it will (should) never happen

* fix linter error

* refactors naming to overrides service

checks quotas in separate function

* fix quotas naming

* fixes more quotas -> overrides naming

* use logger from ctx

* linter - remove trailing whitespace

* log FromContext() when checking quotas

* adds events to spans instead of create new spans

updates tenant -> namespace naming

few other minor fixes
2025-11-27 10:29:16 -06:00
Rafael Bortolon Paulovic
4c869a21a4 feat(unified): data migration integration tests (#114418)
* feat: unified storage migrations integration tests

* chore: add comment and adjust db path name

* chore: refactor test cases into interface
2025-11-27 13:35:49 +01:00
owensmallwood
f116539541 Unified Storage: Update readme (#114415)
* update readme

* Adds message about creating database

* update sample storage config
2025-11-26 15:53:00 -06:00
Matheus Macabu
21c1d9aedd Secrets: Remove unused methods and dependencies from secure value service (#114467) 2025-11-26 12:58:00 +01:00
Misi
93ec32dd6a IAM: Add teams/{id}/groups as a custom endpoint to Teams API (#114228)
* Add teams/{id}/groups as a custom endpoint

* TeamGroupsHandler OSS and Ent registration

* Update OpenAPI spec

* Add indexer tests for external group mapping

* Remove noopsearch

* Remove unnecessary interface declaration, fixes

* Chores

* fmt

* Rename constant

* Align the rest to the changes of main

* Update workspace

* Add missing file
2025-11-25 14:19:57 +01:00
Bruno
9091ac6f5c Secrets: add basic namespace and name checks to keeper store and secure value store (#114355) 2025-11-25 10:04:43 -03:00
Steve Simpson
d50e2a94a3 Alerting: Implement setting of version message in convert API. (#114331)
Part 2 of adding version messages to the `alert_rule_version` able. This allows
setting the message via a header when using the Prometheus conversion API,
which can be useful for e.g. linking changes back to source control.
2025-11-25 11:20:28 +01:00
Jean-Philippe Quéméner
b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Mustafa Sencer Özcan
bca58f5626 feat: add folder tree validation for unified data migration (#114342)
* fix: add validator

* fix: revert startup index
2025-11-24 11:02:14 +01:00
João Calisto
c2c443757d Unified Storage: allow rebuilding indexes for resource with a new grpc endpoint (#113748)
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint

* remove log line

* fix trace def

* lint

* fix after rebase

* addressing code review changes

* update with one channel per rebuild request

* other review suggestions

* update with review suggestions

* run mockery generate for MockResourceClient

* update tests

* update tests and lint

* fix test
2025-11-21 16:42:15 +00:00
Matheus Macabu
5e949fc955 Secrets: Fix secure value creation timestamp changing when updating it (#114290) 2025-11-21 16:31:17 +01:00
Bruno
0d67442f1a Secrets manager: create secure value using the active keeper (#114039)
* Secrets manager: create secure value using the active keeper

* SecureValueService.Update: fetch secure value from db to get the keeper

* fix keeper_store_test.go

* SecureValueService: fix bug in update where the current version keeper wasn't being passed to the createNewVersion method

* make gofmt

* remove outdated test

* update TestModel

* undo enterprise_imports changes

* use xkube.Namespace

* migrator: set secret_secure_value.keeper to 'system' when the column is null

* indent cue

* fix tests

* fix enterprise imports

* properly fix enterprise imports

* make update-workspace

* go mod tidy

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-21 11:20:16 -03:00
Peter Štibraný
def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Mustafa Sencer Özcan
af684248b5 feat: add reindex trigger as a migration validation (#114283)
* feat: add search validation

* chore: remove comment

* fix: remove local test

* fix: revert rebuild index portion

* chore: remove unnecessary comments

* refactor: validator

* fix: revert setting
2025-11-21 12:21:19 +01:00
Mustafa Sencer Özcan
30c04ab3fc feat: inject unified data migrations in dual writer (#114138)
* feat: draft changes for on-prem unified migration

* feat: further draft changes for on-prem unified migration

* fix: remove some tbis

* refactor: rename

* fix: another approach

* fix: background service related issues

* fix: address comments

* fix: make gen-go

* fix: background service related issues

* feat: refactor dual writer and legacy migrator

* fix: minor issues

* feat: working version in oss

* fix: wire

* fix: revert test data override

* fix: enterprise related issues

* chore: add todo

* fix: revert dual writer method

* fix: lint

* chore: logger format

* fix: reduce log level

* fix: log change

* fix: disable

* fix: address comments

* fix: return error on dual writer service

* fix: merge conflict

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2025-11-20 16:40:20 +01:00
Rafael Bortolon Paulovic
65fd15bbf9 feat: define unified disable migrations flag (#114120)
chore: add US data migrations config
2025-11-20 15:47:44 +01:00
Peter Štibraný
42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
Mustafa Sencer Özcan
0d4ad01b65 feat: add unified data migrations for dashboard and folders (#113853) 2025-11-19 09:09:08 +01:00
owensmallwood
8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Rafael Bortolon Paulovic
0fc0fb8db5 chore: move enable search to US configuration (#114070) 2025-11-18 17:32:25 +01:00
Peter Štibraný
ec74a721ea Be more tolerant to invalid JSON when parsing dashboards for indexing (#114040)
* Tags can only be string array.

* Be more lenient when parsing dashboard. Parse what we can, don't error out easily.

* Check element before parsing, log unexpected types.
2025-11-18 14:01:57 +01:00
Rafael Bortolon Paulovic
32da63a20f chore: remove US history pruner feature toggle (#114014) 2025-11-17 20:47:37 +01:00
Georges Chaudy
1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Jean-Philippe Quéméner
e92a976de2 fix(unified-storage): process list items concurrently (#113801) 2025-11-13 12:08:14 +01:00
Mustafa Sencer Özcan
6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Mike
6c512dabdc Secrets: Fix MariaDB syntax error due to unsupported CTE syntax (#111610) (#113690)
* Secrets: fix MariaDB syntax error due to unsupported CTE syntax (#111610)

* parametrize guid/created columns and re-generate test fixtures

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-11-12 15:15:43 +01:00
Mustafa Sencer Özcan
735b776edc fix: cleanup legacy resource if it is created in legacy during dual update (#113753) 2025-11-12 15:02:36 +01:00
Jean-Philippe Quéméner
b9e39cdfcc chore(unified-storage): add debug log for read after write (#113747) 2025-11-12 14:39:03 +01:00
Jean-Philippe Quéméner
76ab09a6a2 feat(unified-storage): add tracing to apistore (#113714) 2025-11-12 09:48:56 +01:00
Jean-Philippe Quéméner
589435b7c2 fix(unified-storage): resource server tracing (#113582) 2025-11-07 11:51:32 +01:00
Nathan Vērzemnieks
f4b23253b1 DataSources: Update SDKs in support of auth service (#112101)
* DataSources: Update SDKs for auth service

* Fix deprecated methods & types for new arrow-go version
2025-11-07 10:15:27 +01:00