Commit Graph

36 Commits

Author SHA1 Message Date
Peter Štibraný
da43e2ae07 Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00
owensmallwood
cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Mustafa Sencer Özcan
7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Mustafa Sencer Özcan
4a779c4ccb fix: unified resource server list queries order column (#108094) 2025-07-16 08:09:31 +02:00
Stephanie Hingtgen
7864c1660f K8s: Get trash fixes (#106411) 2025-06-11 13:54:38 -05:00
Matheus Macabu
fc9f32a9f6 SQLTemplates: Add helper to ensure all templates have a test-case (#103964)
* SQLTemplates: Add helper to ensure all templates have a test-case associated

* UnifiedStorage: Add missing sql template test case

* LegacyDashboards: Add sql templates fs to test cases for exhaustiveness check

* RBACStore: Add sql templates fs to test cases for exhaustiveness check

* LegacyIAM: Add missing sql template test cases
2025-04-22 11:21:51 +02:00
Jean-Philippe Quéméner
3380ea441a feat(unified-storage): return guid, group and resource on read/list (#104121) 2025-04-17 12:58:58 +02:00
Ryan McKinley
9c49c601f1 Unistore: Add generation column (#102222) 2025-04-01 20:38:23 +03:00
Mustafa Sencer Özcan
fe1a72e41b fix(unified-storage): unlist the deleted resource's versions (#102588) 2025-03-27 18:34:37 +01:00
Stephanie Hingtgen
c33a53a47a K8s: Remove restore functionality; can be done with list (#102560) 2025-03-20 16:38:32 -05:00
Jean-Philippe Quéméner
3e3ab8e064 fix(unified-storage): reduce lock contention when pruning (#102425) 2025-03-19 12:15:04 +01:00
Jean-Philippe Quéméner
1700a8aa9f feat(unified-storage): prune history table based on limits (#101970) 2025-03-17 11:36:38 +01:00
Georges Chaudy
3589baac68 Unistore: Batch write events (#101381)
* Batch write events

* Improve instrumentation

* Measure batch phases

* Detect lock contention

* remove the execBatch goroutine

* removing tracing prefix

* detect context cancel

* batch channel map
2025-03-13 10:24:12 +02:00
Ryan McKinley
a5355fd66c Storage: Add command line tool to migrate legacy dashboards (and folders) to unified storage (#99199) 2025-02-11 19:57:46 +02:00
Stephanie Hingtgen
0cef2b9ae7 Dashboard Versions: Make compatible with app platform (#99327) 2025-01-28 08:17:52 -06:00
Ryan McKinley
9d30911107 SQL/Storage: Use xorm code migration for DeletionMarker (#99353) 2025-01-22 14:07:08 +03:00
Ryan McKinley
d91d729749 Storage/SQL: Support more filter by namespace/group/resource (#99249) 2025-01-21 12:15:08 +03:00
Ryan McKinley
356b32008b Storage: Show history+trash using the list command (#99009)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-01-17 15:54:25 +03:00
Ryan McKinley
429da7fd68 SQL/Blob: Add support for blob storage to SQL backend (#98192) 2025-01-08 22:08:10 +02:00
Stephanie Hingtgen
8f6e9f8ed0 Restores in app platform (#97582) 2024-12-13 16:55:43 -06:00
Ryan McKinley
ea17b79c09 Search: Return counts for values within an folder/repository (#97534) 2024-12-10 20:37:37 +02:00
Ryan McKinley
8165258a2d Storage: Query stats within a namespace (#97403) 2024-12-05 13:58:13 +03:00
Ryan McKinley
9d89d8757f Search: Build index from resource stats (#97320) 2024-12-03 20:20:27 +03:00
Georges Chaudy
c3571752b6 Unitstore: Fix the Folder migration (#96078)
* UniStore: Add folder to the SQL backend

This reverts commit aaa4f4a23e.

* fix folder migration
2024-11-12 12:52:04 +01:00
Leonor Oliveira
aaa4f4a23e Revert "UniStore: Add folder to the SQL backend" (#95966)
Revert "UniStore: Add folder to the SQL backend (#95671)"

This reverts commit f075662696.
2024-11-06 17:04:57 +02:00
Georges Chaudy
f075662696 UniStore: Add folder to the SQL backend (#95671)
* Add folder to the SQL backend

* remove unused key

* fix tests

* Return folders from watch
2024-11-05 10:37:23 +01:00
Georges Chaudy
d999b415df UniStore: Use epoch with microsecond resolution as RV (#92638)
* Use epoch with microsecond resolution as RV

* fix backend tests

* Add solution for when the clock goes back

* Add solution for when the clock goes back

* generate mocks

* go lint

* remove comment

* Use Greatest instead of max in msyql and postgres

* update tests

* Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* cast to bigint

* add additional round trip

* increment the RV using 2 sql round trips instead of 3

* cleanup comments

* cast unix timestamp to integer

* fix postgres query

* remove old increment test data

* remove greatest

* cast unix_timestamp to signed

* Use statement_timestamp instead of clock_timestamp

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-10-11 12:11:33 +03:00
Georges Chaudy
03f55e5288 Unistore : Ensure Watch works in HA mode (#94120)
* Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)"

This reverts commit 7c3fc2f261.

* make previous_resource_version nullable

* handle nil case
2024-10-07 10:01:53 +02:00
Georges Chaudy
7c3fc2f261 Revert "Unistore : Ensure Watch works in HA mode." (#94097)
Revert "Unistore : Ensure Watch works in HA mode. (#93428)"

This reverts commit 0a26c9e9ae.
2024-10-01 18:45:47 +00:00
Georges Chaudy
0a26c9e9ae Unistore : Ensure Watch works in HA mode. (#93428)
* Replace Watch with WatchNext

* remove watchset

* fix previous page and closing the channel

* Remove the broadcaster cache to prevent dupplicated events

* add watch bookmark

* add watch bookmark

* cleanup comments

* disable the tests for bookmarks for now

* Ensure we send previosu events

* lint

* re-introduce the cache

* load from cache

* disabling legacy test

* disabling legacy test

* Update pkg/storage/unified/resource/server.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* Could not read previous events

* add proper migration

* Add previous_resource_version to both history and resource

* First event should have an RV of 2 and not 1

* Test both storage backends

* fix the inital RV for the sql backend

* ensure graceful stop of the stream decoder

* gocyclo

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-09-30 13:14:07 +02:00
Ryan McKinley
5a30e12a10 SQLTemplate: Make Ident only work for identifiers (not any string) (#92387) 2024-08-27 13:22:40 +03:00
Ryan McKinley
d9cabe5e14 SQL: Add sql template test helper (#91953) 2024-08-16 14:36:56 +03:00
Georges Chaudy
36c2e0cfae unifiedStorage: sort by namespace/name ASC (#91314)
* unified: sort by key asc

* sort by key

* keep tests commented
2024-08-01 08:27:01 +00:00
Ryan McKinley
f804b0baa3 Storage: Return an iterator from backend List (#91185) 2024-07-31 12:05:59 +03:00
Georges Chaudy
7ad37ff4e4 UnifiedStorage: ensure list from history works across all 3 database backends (#91134)
* Ensure List is sorted by resource version DESC

* Ensure List is sorted by resource version DESC

* fix list in the past for postgres
2024-07-29 13:49:54 -06:00
Georges Chaudy
08c611c68b ResourceServer: Resource store sql backend (#90170) 2024-07-18 08:03:18 -07:00