Commit Graph
123 Commits
Author SHA1 Message Date
Jean-Philippe Quéméner 002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Georges Chaudy 15b3de5893 search: add index batching (#104163)
* add basic search backend integration tests

* add search backend benchmark

* add benchmark indexServer

* fix

* lint

* add more tests

* lint

* do not use the poller

* batch write

* refactor and add tests

* improvements

* improvements

* cleanup

* only observe index success

* add monitorIndexEvents method

* nit use switch instead of if

* make newIndexQueueProcessor private

* simplify runProcessor

* go lint
2025-05-09 15:36:21 +02:00
Will Assis 4adebd6058 unified-storage: setup ring to shard requests (#103783)
* Updates the instrumentation_server service to use mux instead of the builtin router, and have it store the router in the module server: this is so we can register the /ring endpoint to check the status of the ring
* Create a new Ring service that depends on the instrumentation server and declares it as a dependency for the storage server
* Create standalone MemberlistKV service for Ring service to use
* Update the storage server Search and GetStats handler to distribute requests if applicable
2025-04-25 13:08:44 -04:00
Jean-Philippe Quéméner 3380ea441a feat(unified-storage): return guid, group and resource on read/list (#104121) 2025-04-17 12:58:58 +02:00
Georges Chaudy e69052a417 unistore: add check when update the folder of a resource (#102699)
* Add check for move folder

* make the server test generic

* address comment
2025-04-14 16:57:40 +02:00
Peter Štibraný be696dd70c unified storage: check for iterator errors after each call to iter.Next. (#102804)
* unified storage: check for iterator errors after each call to iter.Next.
* Extracted test to separate method, add 500 events before listing.
2025-04-11 16:25:40 +02:00
Stephanie Hingtgen 68ed0feeff Unistore: Change to 404 rather than 403 if not found (#103743) 2025-04-09 19:14:39 -05:00
Ryan McKinleyandGeorges Chaudy af8a70bbab K8s/Permissions: Enable a grant-permissions annotation action to set default permissions (#102527)
* create permissions

* add key

* lint

* structure as a delayed callback

* legacy API hook

* merge main

* wired up

* and folders

* watch repos

* missing return statement

* Set the correct permissions

* add TestAfterCreatePermissionCreator

* do not add perms on folder create

* fix tests

* add annotation on create

* lint

* lint

* ensure we set permissions when the FT is disabled

* remove custom folder_storage

* fix lint

* change default

* lint

* lint

* fix: annotation

* ensure permissions are added on folder legacy

* remove folderstorage again

* fix tests

* add FT

* undo change to folder

* dashboard on create

* remove annotation for folder

* fix tests

* fix prepare after rebase

* fix tests

* fix tests

* fix tests

* lint

* address comments

* add test for prepareObjectForStorage

* add again skipIfMode as per comment

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-04-09 13:05:37 +02:00
Mariell Hoversholm f0a6327edc Unified Storage: Don't read before create (#102906)
* Unified Storage: Don't read before create

* test: use the existing test infra

* fix: support pq

We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.

* feat: support CDK backend

* revert: there is a postgres_tests block

* fix(CDK): only check existence on ADDED updates

* fix(CDK): use ReadResource to deal with deleted files
2025-03-31 15:06:31 +02:00
Stephanie Hingtgen 0087d7bd58 Unistore: Skip label for getting full path (#102928) 2025-03-26 15:50:53 -05:00
Mariell Hoversholm f7b9f1ce69 Unified Storage: Return an already exists error (#102857)
* Unified Storage: Return an already exists error

When inserting a resource that already exists (i.e. race condition), we can safely catch UNIQUE constraint violations
and transform them into a `k8s.io/apimachinery/pkg/api/errors` error that stands the test of `errors.IsAlreadyExists`.

* feat: clarify existing conflict error

* chore: make update-workspace

* feat: make new package for backend error

* fix: assign dependency owner

* feat: use dialect for checking error type

* chore: go generate

* revert: to 5af369166d
2025-03-26 14:44:44 +01:00
6d570db312 Storage: Add ascending order support for NotOlderThan queries and introduce ResourceVersionMatch_Unset as default (#102505)
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.

* Change protobuf order

* Make backwards compatible

* Update pkg/storage/unified/sql/backend.go

Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>

---------

Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
2025-03-21 18:35:32 +02:00
Stephanie Hingtgen c33a53a47a K8s: Remove restore functionality; can be done with list (#102560) 2025-03-20 16:38:32 -05:00
Will Assis 31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
Ryan McKinley 64f1df69d5 Search: Rename managed object index (#101856) 2025-03-10 19:48:53 +03:00
Ryan McKinleyandStephanie Hingtgen dc2defd84f K8s/Annotations: Use manager/source annotations rather than repo (#101313)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-03-05 08:54:20 +02:00
Will AssisandJean-Philippe Quéméner f5e5824bab fix (unified-storage): stop registering unified storage metrics in global state (#101322)
* move prometheus.register for unified storage metrics into metrics.go and do most of the plumbing to get it to work

* convert StorageApiMetrics to pointer and check for nil before using it

* rename type and variables to something more sensible

---------

Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
2025-02-28 07:39:39 -05:00
Ryan McKinley 806c043e45 UnifiedStorage: Rename Batch processing to Bulk (#101413) 2025-02-28 08:41:08 +03:00
Ryan McKinley 8f37822050 Unified/Blob: Allow direct access to UIDs (#101339) 2025-02-26 16:18:59 +02:00
Karl Persson fa74d1c36d Authn: Sync authlib and update how we construct authn client interceptor (#101124)
* Sync authlib and update how we construct authn client interceptor

* Remove namespace from checker
2025-02-26 09:22:09 +01:00
Georges Chaudy 53e91fd5e8 unistore: close event stream on context cancelation (#101293)
* add tests for broacaster

* fix sql notifier not closing the stream

* fix sql notifier not closing the stream

* close sub

* fix broadcaster test

* fix broadcaster test

* suggestion
2025-02-25 08:28:31 -08:00
Georges Chaudy e13bd52da6 unistore: move continue to the resource package (#101206)
* minor improvements to tests

* move continue token

* update sql backend
2025-02-24 18:02:30 +02:00
Georges Chaudy b850c9fa68 unistore: Add missing verb to compile (#100972)
Add missing verb to compile
2025-02-19 17:06:26 +02:00
Ryan McKinley a5355fd66c Storage: Add command line tool to migrate legacy dashboards (and folders) to unified storage (#99199) 2025-02-11 19:57:46 +02:00
Stephanie Hingtgen 0cef2b9ae7 Dashboard Versions: Make compatible with app platform (#99327) 2025-01-28 08:17:52 -06:00
437b7a565d Auth: Add access token to in-proc communication and ServiceIdentity (#98926)
Use fake access token for in-proc grpc and add ServiceIdentity 
---------

Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Karl Persson <23356117+kalleep@users.noreply.github.com>
2025-01-24 14:03:23 +01:00
Ryan McKinley 680e6bc1f8 Authlib: Use types package rather than claims (#99243) 2025-01-21 12:06:55 +03:00
Ryan McKinleyandStephanie Hingtgen 356b32008b Storage: Show history+trash using the list command (#99009)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-01-17 15:54:25 +03:00
Georges Chaudy 98e9f3a534 [unistore] wire authlib compile (#99027)
* Wire authz client compile method

* add verb to metric

* remove tipo
2025-01-16 14:11:55 +01:00
Ryan McKinley cd46f1ddb9 Search: Remove history query (#99026) 2025-01-15 12:49:47 -06:00
Karl Persson 3f71a72c1a Authz: Remove "wrapper" interface and only check feature toggle for grpc mode (#98933)
* Remove "wrapper" interface and only check feature toggle for grpc and cloud mode

* Only set name for update checks

* Set dashboard permissions for admin user
2025-01-15 09:23:56 +01:00
Ryan McKinley ed39259461 Search: Replace Origin calls with Repository (#98754) 2025-01-10 20:27:10 +02:00
Ryan McKinley 429da7fd68 SQL/Blob: Add support for blob storage to SQL backend (#98192) 2025-01-08 22:08:10 +02:00
Stephanie Hingtgen 8f6e9f8ed0 Restores in app platform (#97582) 2024-12-13 16:55:43 -06:00
Ryan McKinley ea17b79c09 Search: Return counts for values within an folder/repository (#97534) 2024-12-10 20:37:37 +02:00
owensmallwood d762a96436 Unified Storage: Init at startup, fix traces, and speed up indexing (#97529)
* dont lazy init unified storage

* Inits index when creating new resource server. Fixes trace propagation by passing span ctx. Update some logging.

* Use finer grained cache locking when building indexes to speed things up. Locking the whole function was slowing things down.

* formatting

* linter fix

* go mod

* make update-workspace

* fix workspaces check error

* update dependency owner in mod file

* wait 1 second before querying metrics

* try with big timeout, see if fixes CI. Wont fail locally.

* skips postgres integration test. Only fails in drone. Will fix later.

* put delay back to 500 ms
2024-12-09 22:32:19 -06:00
Ryan McKinley 8165258a2d Storage: Query stats within a namespace (#97403) 2024-12-05 13:58:13 +03:00
Ryan McKinley 9d89d8757f Search: Build index from resource stats (#97320) 2024-12-03 20:20:27 +03:00
Ryan McKinley f6ccf976e5 UnifiedSearch: Use ResourceIndex from dashboards apiserver (v0alpha1 only) (#96939) 2024-11-27 07:57:53 +02:00
Ryan McKinley 94262fd095 Search: expose search on dashboard apiserver (v0alpha1) (#96907) 2024-11-22 22:26:56 +02:00
Ryan McKinley 0cb6c3d7bf UnifiedSearch: Introduce a DocumentBuilder interface (#96738) 2024-11-21 07:53:25 +02:00
Ryan McKinley cc6d057a18 Provisioning: Rename k8s origin metadata to repo (#96524) 2024-11-15 17:26:14 +03:00
Ryan McKinley d9b87ef987 Storage: Add authz access client stubs (#95548)
authz stubs

authz stubs

lint fix

Add folder

move check.Name on creat

fix watch
2024-11-13 13:17:15 +01:00
Ryan McKinley e9956f2345 Storage: Always set a UID in the apiserver (#96086) 2024-11-12 18:58:32 +02:00
Georges Chaudy c3571752b6 Unitstore: Fix the Folder migration (#96078)
* UniStore: Add folder to the SQL backend

This reverts commit aaa4f4a23e.

* fix folder migration
2024-11-12 12:52:04 +01:00
Ryan McKinley c0de407fee K8s/Dashboards: Delegate large objects to blob store (#94943) 2024-11-09 08:09:46 +03:00
owensmallwood b6596db75e Unified Storage Indexer: Build tenant indexes concurrently (#95795)
* WIP - build tenant indexes concurrently

* adds local dev seeders and readme for indexer

* update logging and adds locking in getShard()

* update logs

* Adds Namespaces func. Initializes index after ResourceServer is created.

* fixes Count() and updates test lint issues

* Initialize index separately. Don't do it when creating the resource server. Makes testing really awkward.

* fix lint error

* handles error when getting namespaces

* updates test and index helper funcs
2024-11-06 12:58:07 -06:00
Leonor Oliveira aaa4f4a23e Revert "UniStore: Add folder to the SQL backend" (#95966)
Revert "UniStore: Add folder to the SQL backend (#95671)"

This reverts commit f075662696.
2024-11-06 17:04:57 +02:00
Georges Chaudy f075662696 UniStore: Add folder to the SQL backend (#95671)
* Add folder to the SQL backend

* remove unused key

* fix tests

* Return folders from watch
2024-11-05 10:37:23 +01:00
owensmallwood 5e3e7cca0e Search PoC: Adds traces to indexer (#95372)
* adds traces to indexer

* fix linting errors
2024-10-25 00:05:33 +03:00