Commit Graph
38 Commits
Author SHA1 Message Date
Will Assis 755b479be4 unified-storage: make sql backend update key_path for kv store (#114879)
* unified-storage: update resource_history_update_rv.sql to populate key_path in resource_history
2025-12-10 07:06:06 -05:00
Will AssisandGeorges Chaudy 047e6d45fa unified-storage: use name instead of offset in kvstore continue token (#113560)
* unified-storage: use name instead of offset in kvstore continue token

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-12-02 10:35:03 -05:00
owensmallwood 8dddff3ce4 Unified Storage: Pass ns, group, resource to GetResourceStats instead of just namespace (#114050)
* passes nsr to GetResourceStats instead of just namespace

* removes ns check

* fixes failing tests

* make update-workspace

* pass group and resource from rebuild request when getting resource stats
2025-11-18 13:05:21 -06:00
Georges Chaudy 1162fa5104 kvstore: Add optimistic locking for unified resource storage backend (#113230)
* Add optimistic concurrency

* add optimistic concurrency

* fix test

* nit

* fix tests for sql

* fix tests for sql

* rebase fix

* add one more check

* Implement GetLatestAndPredecessor method in datastore and add corresponding tests. This new functionality retrieves the latest resource version and its immediate predecessor, handling cases for single and non-existent resources. Update WriteEvent to utilize this method for improved optimistic concurrency control.

* Enhance optimistic concurrency control in WriteEvent method. Added checks for concurrent create operations to ensure only one succeeds, preventing race conditions. Updated tests to validate this behavior with multiple concurrent create attempts.

* lint

* Refactor optimistic concurrency check in WriteEvent method. Simplified the logic by removing unnecessary condition for single version existence, ensuring more robust handling of concurrent modifications.
2025-11-14 10:42:39 +01:00
Will Assis c9e4c26c11 unified-storage: add more list pagination tests (#113543)
* unified-storage: add more list pagination tests
2025-11-06 13:36:02 -05:00
Georges Chaudy ddeb970b4d unistore: replace CDK backend with KV store backend (again) (#113184)
* Reapply "unistore: replace CDK backend with KV store backend"" (#113132)

This reverts commit 7127b2538c.

* enable cluster scope
2025-11-04 00:33:36 +00:00
Stephanie Hingtgen 7127b2538c Revert "unistore: replace CDK backend with KV store backend"" (#113132)
Revert "unistore: replace CDK backend with KV store backend" (#112746)"

This reverts commit fe9c21ebf8.
2025-10-28 21:37:51 +00:00
Georges Chaudy fe9c21ebf8 unistore: replace CDK backend with KV store backend" (#112746)
* deprecate the cdk backend in favor of the kv storage backend

* lint

* fix watchtests

* cover limit=0 for ListHistory

* fix rv too large
2025-10-28 10:56:08 +01:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Ryan McKinley c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Ryan McKinley cfbf64c3fd Validation: Move validation into apimachinery package (#111736) 2025-09-30 12:59:33 +00:00
Alexander ZobninandJo Garnier 294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
Peter Štibraný 238f121e10 LastModifiedSince: return latestRV even when it hasn't changed. (#110391)
Return latestRV.
2025-09-01 08:45:40 +00:00
Peter Štibraný 3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
owensmallwood 699bffc9e4 Unified Storage: Dont need to test casing with ordering (#110245)
Dont need to test casing with ordering.
2025-08-27 14:06:43 -06:00
owensmallwood b4ed217656 Unified Storage: Removes check for name ordering and adds test (#110183)
removes check for name ordering. Adds test to assert expected name and rv ordering.
2025-08-26 12:39:30 -06:00
Matheus Macabu 78ac555c4d CI: Run gofmt on Linting action (#110159)
* CI: Run gofmt on Linting action

* Chore: Run gofmt on all packages
2025-08-26 16:14:03 +02:00
owensmallwood cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Mustafa Sencer Özcan 7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Mustafa Sencer Özcan 4a779c4ccb fix: unified resource server list queries order column (#108094) 2025-07-16 08:09:31 +02:00
Georges Chaudy b6dd08da2f unistore: fix delete and db closed in kv store (#107918)
* fix delete and db closed

* fix tests
2025-07-10 11:34:36 +02:00
Georges Chaudy d55541735a unistore: use different ns for each test (#107509)
use different ns for each test
2025-07-02 13:50:30 +02:00
Georges ChaudyandPeter Štibraný 696657bdd1 unistore: add kv based storage backend (#107305)
* Add datastore

* too many slashes

* lint

* add metadata store

* simplify meta

* Add eventstore

* golint

* lint

* Add datastore

* too many slashes

* lint

* pr comments

* extract ParseKey

* readcloser

* remove get prefix

* use dedicated keys

* parsekey

* sameresource

* unrelated

* name

* renmae tests

* add key validation

* fix tests

* refactor a bit

* lint

* allow empty ns

* get keys instead of list

* rename the functions

* refactor yield candidate

* update test

* unistore: add LastResourceVersion to datastore

* lint

* use map string

* missing err check

* fix

* Add storage backend

* remove hasmore

* fix tests

* small refactor

* pre-alloc

* extract the folder

* lint

* refactor

* handle context canceled in ListHistory to pass the tests

* fix the resource test

* unistore: provide generic tests for the kv interface (#107443)

unistore: move the kv tests to the testing package

* Update pkg/storage/unified/resource/storage_backend_test.go

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>

* address comments

* comments

* comments

* comments

* normalise the names and add helper method

* events comments

* rename function

---------

Co-authored-by: Peter Štibraný <pstibrany@gmail.com>
2025-07-02 10:57:37 +00:00
Stephanie Hingtgen 7864c1660f K8s: Get trash fixes (#106411) 2025-06-11 13:54:38 -05:00
Jean-Philippe Quéméner a13fe75de8 refactor(unified-storage): set the GUID in the resource server (#105683) 2025-05-21 09:49:49 +02:00
Jean-Philippe Quéméner 002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Georges Chaudy e162c69c34 search: fix document missing at startup (#105198)
* fix document missing at startup

* go-lint

* fix tests

* fix tests

* fix integration tests now that we are storing real values
2025-05-12 07:56:25 +02:00
Marco de AbreuandMarco de Abreu b3b92e2471 Storage: Change timeout for flaky test (#104065)
Change timeout for flaky test

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
2025-04-15 19:28:27 +01:00
Peter Štibraný be696dd70c unified storage: check for iterator errors after each call to iter.Next. (#102804)
* unified storage: check for iterator errors after each call to iter.Next.
* Extracted test to separate method, add 500 events before listing.
2025-04-11 16:25:40 +02:00
Mariell Hoversholm f0a6327edc Unified Storage: Don't read before create (#102906)
* Unified Storage: Don't read before create

* test: use the existing test infra

* fix: support pq

We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.

* feat: support CDK backend

* revert: there is a postgres_tests block

* fix(CDK): only check existence on ADDED updates

* fix(CDK): use ReadResource to deal with deleted files
2025-03-31 15:06:31 +02:00
Mustafa Sencer Özcan fe1a72e41b fix(unified-storage): unlist the deleted resource's versions (#102588) 2025-03-27 18:34:37 +01:00
6d570db312 Storage: Add ascending order support for NotOlderThan queries and introduce ResourceVersionMatch_Unset as default (#102505)
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.

* Change protobuf order

* Make backwards compatible

* Update pkg/storage/unified/sql/backend.go

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2025-03-21 18:35:32 +02:00
Georges Chaudy 0bafd4e99d unistore: update spanner ddl to include unistore tables (#102198)
* provision unistore tables

* update the tests to use generated namespace

* update ddl

* update ddl

* clean streams

* missing space

* use engine.db.query
2025-03-18 14:37:11 +01:00
Ryan McKinley 8f37822050 Unified/Blob: Allow direct access to UIDs (#101339) 2025-02-26 16:18:59 +02:00
Georges Chaudy 53e91fd5e8 unistore: close event stream on context cancelation (#101293)
* add tests for broacaster

* fix sql notifier not closing the stream

* fix sql notifier not closing the stream

* close sub

* fix broadcaster test

* fix broadcaster test

* suggestion
2025-02-25 08:28:31 -08:00
Georges Chaudy e13bd52da6 unistore: move continue to the resource package (#101206)
* minor improvements to tests

* move continue token

* update sql backend
2025-02-24 18:02:30 +02:00
Georges Chaudy 40d1677265 unistore: improve storage backend integration tests (#101134)
* Add Stats tests

* Add integration tests for getstats

* fix signature
2025-02-21 15:06:23 +01:00
Georges Chaudy 6e5de80ef2 unistore: refactor the storage backend test suite (#101061)
* refactor the storage backend test suite

* move testing

* pass a newBackendFunc
2025-02-20 16:43:26 +01:00