Commit Graph

121 Commits

Author SHA1 Message Date
Jean-Philippe Quéméner 08994304d8 fix(unified-storage): check resource_api cfg for isHA (#102283) 2025-03-18 12:33:27 +01:00
Jean-Philippe Quéméner 1700a8aa9f feat(unified-storage): prune history table based on limits (#101970) 2025-03-17 11:36:38 +01:00
Peter Štibraný 6f12712ad6 Add Spanner dialect to pkg/storage/unified/sql/sqltemplate package. (#102121) 2025-03-13 16:38:05 +01:00
Will Assis 31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
Georges Chaudy 3589baac68 Unistore: Batch write events (#101381)
* Batch write events

* Improve instrumentation

* Measure batch phases

* Detect lock contention

* remove the execBatch goroutine

* removing tracing prefix

* detect context cancel

* batch channel map
2025-03-13 10:24:12 +02:00
Ryan McKinley 64f1df69d5 Search: Rename managed object index (#101856) 2025-03-10 19:48:53 +03:00
linoman b7a0aeeb0d SCIM: Disable auto assign organization if the user has been provisioned (#101307)
* Add isProvisioned field to model

* Add new isProvisioned column to migration

* Disable auto assignment to organization if the user is provisioned

* add annotation to user model

* add annotation to user models

* Remove IsProvisioned field from Identity

* Move new field assignenment and add default value

* Update annotations for user query results

* Remove isProvisioned from identity

* Add new column to test

* Resolve user from identity at SyncOrgHook
2025-03-03 17:51:23 +01:00
Will Assis f5e5824bab fix (unified-storage): stop registering unified storage metrics in global state (#101322)
* move prometheus.register for unified storage metrics into metrics.go and do most of the plumbing to get it to work

* convert StorageApiMetrics to pointer and check for nil before using it

* rename type and variables to something more sensible

---------

Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
2025-02-28 07:39:39 -05:00
Ryan McKinley 806c043e45 UnifiedStorage: Rename Batch processing to Bulk (#101413) 2025-02-28 08:41:08 +03:00
Georges Chaudy 8c935c8f4a [unistore] Add benchmark for write throughput (#101345)
* Add generic benchmark

* address comments
2025-02-26 17:17:35 +02:00
Ryan McKinley 8f37822050 Unified/Blob: Allow direct access to UIDs (#101339) 2025-02-26 16:18:59 +02:00
Karl Persson fa74d1c36d Authn: Sync authlib and update how we construct authn client interceptor (#101124)
* Sync authlib and update how we construct authn client interceptor

* Remove namespace from checker
2025-02-26 09:22:09 +01:00
Georges Chaudy 53e91fd5e8 unistore: close event stream on context cancelation (#101293)
* add tests for broacaster

* fix sql notifier not closing the stream

* fix sql notifier not closing the stream

* close sub

* fix broadcaster test

* fix broadcaster test

* suggestion
2025-02-25 08:28:31 -08:00
Jean-Philippe Quéméner 9282e4cad4 fix(unified-storage): don't use in-process if db type is not set (#101254) 2025-02-24 21:44:42 +01:00
Georges Chaudy e13bd52da6 unistore: move continue to the resource package (#101206)
* minor improvements to tests

* move continue token

* update sql backend
2025-02-24 18:02:30 +02:00
Georges Chaudy 40d1677265 unistore: improve storage backend integration tests (#101134)
* Add Stats tests

* Add integration tests for getstats

* fix signature
2025-02-21 15:06:23 +01:00
Jean-Philippe Quéméner 7be1fd953a fix(unified-storage): enable in-process events for single-instance (#100807) 2025-02-21 12:25:35 +01:00
Peter Štibraný b4c4b9abbd Remove embedded WriteEvent from WrittenEvent struct, to avoid having unnecessary fields. (#101081) 2025-02-20 17:34:25 +01:00
Georges Chaudy 6e5de80ef2 unistore: refactor the storage backend test suite (#101061)
* refactor the storage backend test suite

* move testing

* pass a newBackendFunc
2025-02-20 16:43:26 +01:00
Jean-Philippe Quéméner 4a8b1ba61b tests(unified-storage): use unified storage grpc client (#101063) 2025-02-20 15:03:08 +01:00
Karl Persson 6eeb28e312 Authn: use authenticator for grpc (#99573)
* Remove usage of grpc-authenticator

* Cleanup client construction code
2025-02-17 10:58:59 +01:00
Peter Štibraný d0394bfa7e Extract NewSearchOptions from unified sql client setup. (#100719)
* Extract NewSearchOptions from unified sql client setup.

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-02-14 16:39:48 +01:00
Jean-Philippe Quéméner c522a5b13b fix(unified-storage): proper setup TLS in new db_engine for MySQL (#100686) 2025-02-14 16:23:25 +01:00
Georges Chaudy b1222be02e unistore: add small buffer of watched events (#100431)
* change log level

* Add a small buffer when watching events
2025-02-14 12:18:59 +02:00
owensmallwood a34e7e176d Unified Storage: Sprinkles latency metric (#100542)
* add sprinkles latency metric

* fixes failing tests - forgot to register metric only once
2025-02-12 16:22:18 -06:00
Ryan McKinley a5355fd66c Storage: Add command line tool to migrate legacy dashboards (and folders) to unified storage (#99199) 2025-02-11 19:57:46 +02:00
owensmallwood 3ba0d8d4b5 Unified Storage: Improve observability for indexing latency (#99700)
* adds extra debug logs and a new metric for poller query latency to help us better understand watch and index latency for write events

* adds trace span to the index for handling index write events
2025-01-28 12:30:20 -06:00
Stephanie Hingtgen 0cef2b9ae7 Dashboard Versions: Make compatible with app platform (#99327) 2025-01-28 08:17:52 -06:00
Misi 437b7a565d Auth: Add access token to in-proc communication and ServiceIdentity (#98926)
Use fake access token for in-proc grpc and add ServiceIdentity 
---------

Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Karl Persson <23356117+kalleep@users.noreply.github.com>
2025-01-24 14:03:23 +01:00
Ryan McKinley c4c934e0bd SQL/Storage: Remove SkipDataMigration flag (#99404) 2025-01-23 12:34:48 +02:00
owensmallwood dd483fc17f Unified Storage: Search permissions put behind feature flag (#99340)
* add feature flag

* puts search permission filtering behind a feature flag

* fixes issue with doc match id. When the match is for an in-memory index, the internal id is a string (this is what we expected). However, when its a file-based index, the internal id is a binary encoded int64 that point to something internally. So to get the id, we need to use ExternalID() instead of relying on the indexInternalID to be the correct format.

* adds debug log

* update comment

* formatting
2025-01-22 05:38:37 -06:00
Ryan McKinley 9d30911107 SQL/Storage: Use xorm code migration for DeletionMarker (#99353) 2025-01-22 14:07:08 +03:00
Ryan McKinley d91d729749 Storage/SQL: Support more filter by namespace/group/resource (#99249) 2025-01-21 12:15:08 +03:00
Ryan McKinley 680e6bc1f8 Authlib: Use types package rather than claims (#99243) 2025-01-21 12:06:55 +03:00
Ryan McKinley 356b32008b Storage: Show history+trash using the list command (#99009)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-01-17 15:54:25 +03:00
Gabriel MABILLE 70ddf9cb76 Authenticator: Return gRPC errors (#99000) 2025-01-16 14:33:57 +01:00
Georges Chaudy 98e9f3a534 [unistore] wire authlib compile (#99027)
* Wire authz client compile method

* add verb to metric

* remove tipo
2025-01-16 14:11:55 +01:00
Ryan McKinley cd46f1ddb9 Search: Remove history query (#99026) 2025-01-15 12:49:47 -06:00
Karl Persson 3f71a72c1a Authz: Remove "wrapper" interface and only check feature toggle for grpc mode (#98933)
* Remove "wrapper" interface and only check feature toggle for grpc and cloud mode

* Only set name for update checks

* Set dashboard permissions for admin user
2025-01-15 09:23:56 +01:00
Ryan McKinley ed39259461 Search: Replace Origin calls with Repository (#98754) 2025-01-10 20:27:10 +02:00
Ryan McKinley 429da7fd68 SQL/Blob: Add support for blob storage to SQL backend (#98192) 2025-01-08 22:08:10 +02:00
Ryan McKinley 399cbf7c50 Search: Use new folder when creating a bleve index (#98260) 2024-12-19 18:40:04 +02:00
owensmallwood 4837585cab Unified Storage Search: Add sprinkles (#97820)
* Wire up sprinkles to oss and enterprise. Fetching sprinkles not implemented yet.

* Adds wireset for initializing document builders. Had to init it when creating the service to avoid cyclical imports.

* updates to int64 for stats

* adds config for sprinklesApiServer and gets sprinkles from there when its present

* add comment for later

* adds feature toggle for sprinkles. returns empty results when flag not enabled.

* adds unified storage config setting for sprinkles apiserver page limit

* fixes bug where dashboard uid was not getting set

* when creating dashboard summary, use metadata.name as the dashboard uid

* cleans up wire. use existing oss and enterprise sets to generate doc builders

* remove old wireset

* fix linter - adds missing arg for doc builders

* update dashboard stats in tests

* updates test-data dashboards

* log a warning instead of returning an error if we can't get sprinkles for a namespace

* dont read uid from dashboard json
2024-12-18 11:00:54 -06:00
Stephanie Hingtgen 8f6e9f8ed0 Restores in app platform (#97582) 2024-12-13 16:55:43 -06:00
Ryan McKinley ea17b79c09 Search: Return counts for values within an folder/repository (#97534) 2024-12-10 20:37:37 +02:00
owensmallwood d762a96436 Unified Storage: Init at startup, fix traces, and speed up indexing (#97529)
* dont lazy init unified storage

* Inits index when creating new resource server. Fixes trace propagation by passing span ctx. Update some logging.

* Use finer grained cache locking when building indexes to speed things up. Locking the whole function was slowing things down.

* formatting

* linter fix

* go mod

* make update-workspace

* fix workspaces check error

* update dependency owner in mod file

* wait 1 second before querying metrics

* try with big timeout, see if fixes CI. Wont fail locally.

* skips postgres integration test. Only fails in drone. Will fix later.

* put delay back to 500 ms
2024-12-09 22:32:19 -06:00
owensmallwood 39aede9802 Unified Storage: When prom registry is nil, use the default registry (#97491)
when prom registry is nil, use the default registry
2024-12-05 18:44:58 +02:00
Ryan McKinley 8165258a2d Storage: Query stats within a namespace (#97403) 2024-12-05 13:58:13 +03:00
owensmallwood 70c9c3889f Unified Storage Indexer: Add back metrics (#97310)
* Adds back indexer metrics. Uses config values instead of hardcoded ones.

* cast to int64

* remove unused func

* Index metrics impl doesn't depend on Bleve. Adds a TotalDocs func to SearchBackend interface.

* adds config setting for index_min_count

* rename arg

* rename metric label to namespace instead of slug

* adds default "do nothing" case to satisfy linter

* moves bleve index metrics to search package

* make bleve backend private, dont need to pass in prom reg

* imports

* adds bleve metrics to resource package to avoid circular deps
2024-12-04 15:02:40 -06:00
owensmallwood 9642eedce6 Unified Storage: Use tls preferred when grafana db using ssl (#97378)
When unified storage is using the grafana db (default case for on-prem), if the db is using ssl then set tls=preferred. This fixes an on-prem bug in 11.3.1.
2024-12-03 23:56:13 +02:00