Commit Graph

696 Commits

Author SHA1 Message Date
Peter Štibraný
357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis
33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter Štibraný
f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
maicon
b3ed600bca Revert "Unistore: Add validation for resource names" (#111408)
Revert "Unistore: Add validation for resource names (#110990)"

This reverts commit b63e3fd3ae.
2025-09-19 18:03:23 +00:00
Daniele Stefano Ferru
e69cc03ef9 Secret: add ability to configure extra owner decrypters (#111301)
---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-19 07:41:56 -05:00
maicon
b63e3fd3ae Unistore: Add validation for resource names (#110990)
* Unistore: Add validation for resource names

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-19 09:37:17 -03:00
Alexander Zobnin
72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Ryan McKinley
9743e0c903 DualWriter: Error when folder+dashboard modes do not match when running mode4+ (#111286) 2025-09-18 16:43:39 +03:00
Peter Štibraný
48250a1605 Introduced options to ignore old indexes based on build time and build version. (#111168) 2025-09-17 10:59:59 +02:00
Mustafa Sencer Özcan
4c2240dcc3 fix: introduce an init lock for the storage backend integration tests (#111166) 2025-09-16 14:18:06 +00:00
Will Assis
6b2fb782c8 fix (unified-storage): search returning empty results when query includes small term (#111140)
* fix search not returning anything when a word in the query has less than 3 characters

* fix test name

* fmt

* remove trimspace
2025-09-16 08:55:23 -04:00
Ryan McKinley
0a79b3bdc5 Chore: Upgrade k8s.io/api v0.34.1 and grafana-app-sdk v0.43.1 (#111009) 2025-09-16 13:35:20 +03:00
Georges Chaudy
c251ebf4d5 kvstore: merge the metadata store into the datastore (#110334)
* migrate eventstore to datastore

* Add folder to event key

* lint

* lint

* lint

* lint

* remove foundkye

* refactor the Keys methods to move the Sort outside the ListKey method

* remove bad import

* fix missing params

* lint

* fix test

* perf improvement
2025-09-16 12:16:29 +02:00
Peter Štibraný
f77e99d965 Store build time and build version into index. (#111010) 2025-09-16 10:52:30 +02:00
Alexander Zobnin
294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
owensmallwood
7ce971cba1 Unified Storage: Adds pruner for kv eventstore (#110785)
* Adds pruner for eventstore - default 24 hours. Adds tests.

* update comment

* remove delay on startup. formatting

* updates log message type and removes useless comment

* caller handles goroutine for runCleanupOldEvents()

* simplify timestamp extraction

* adds config for event pruning interval

* uses start and end key to get all expired events

* remove sort when listing keys in event pruner - order doesnt matter

* use snowflake constants

* log when we delete 0 rows

* pass time.Time to cleanup old events func
2025-09-12 14:40:16 -06:00
owensmallwood
310893292f Unified Storage: Add sort order to keys func in datastore (#110714)
* Add sort order to keys func in datastore. Add test to not prune deleted events.

* include sort field in the ListRequestKey instead of it being a separate param
2025-09-11 14:12:30 -06:00
Mustafa Sencer Özcan
2bd76d4044 fix: bump context deadline for integration tests while backend initialization (#110947) 2025-09-11 17:25:13 +02:00
Peter Štibraný
c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný
6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Mustafa Sencer Özcan
b8b85fbf47 fix: add intrumentation for auth server grpc client (#110875) 2025-09-10 17:33:21 +02:00
Stephanie Hingtgen
323738d191 Provisioning: Fix check of who can update (#110835) 2025-09-10 09:04:10 +03:00
Ryan McKinley
7e8bbd2ec4 DualWrite: Avoid dynamic wrapper when mode5 is configured (#110823) 2025-09-09 19:38:33 +03:00
Peter Štibraný
b3cc317cbc unified-storage search: Return bad request error for incomplete request. (#110802)
Return bad request error for incomplete request.
2025-09-09 14:15:45 +02:00
Peter Štibraný
7fd9ab9481 Replace check for integration tests. (#110707)
* Replace check for integration tests.
* Revert changes in pkg/tsdb/mysql packages.
* Fix formatting of few tests.
2025-09-08 15:49:49 +02:00
Peter Štibraný
d09708fe55 Move SkipIntegrationTestInShortMode to testutil. (#110750)
* Move SkipIntegrationTestInShortMode to testutil.

* make update-workspace
2025-09-08 12:50:31 +02:00
Will Assis
005da25698 fix: ListManagedObjects and CountManagedObjects panic when search is not configured (#110726)
fix ListManagedObjects and CountManagedObjects panicking when search index not configured
2025-09-05 20:58:12 +00:00
owensmallwood
dc1c5a610c Unified Storage: Fix broken tests (#110710)
fix broken tests
2025-09-05 17:19:13 +00:00
owensmallwood
d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Bruno
9a641c651f secrets: update test to accept []byte(nil) and []byte{} (#110630)
Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
2025-09-05 10:19:57 -03:00
Peter Štibraný
bd529226a3 unified-storage search: return results matching all search terms (#110672)
* Extract common indexing code.

* Extract common search code.

* Match all terms in the search query.
2025-09-05 11:00:54 +00:00
Ryan McKinley
eeb940e733 Chore: Replace hand crafted mocks with mockery (#110627) 2025-09-05 07:13:15 +00:00
Will Assis
ea7c370edd unified-storage: add ListSinceModified to kv store (#110250)
* implement ListKeysSince in event store

 implement data store version

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-09-04 12:26:40 -04:00
Bruno
6b5cacfade use standard sql in secure_value_lease_inactive.sql (#110532)
* use standard sql in secure_value_lease_inactive.sql

* ci
2025-09-04 10:01:05 -03:00
Peter Štibraný
642d43ff49 search-after-write: Reuse index even if docs count doesn't match (#110527)
* search-after-write: Reuse index even if docs count doesn't match

* Revert unrelated change.
2025-09-03 15:35:46 +00:00
maicon
fa20755d4b fix: Synchronously notify subscribers when writing Create events to CDK backend (#110476)
* Remove goroutine when writing events on CDK backend

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-03 11:49:04 -03:00
Georges Chaudy
1758fa8fbb unistore: use scorch indexing instead of upside_down (#110463)
* use scorchwith in memory

* comments

* refactor
2025-09-03 11:06:08 +02:00
Serge Zaitsev
cdd7a2cfd2 Chore: Disable CGo in tests (#108764)
* make cgo optional for sqlite

* update go.mod; check error code differently

* reduce api surface even more

* move test errors into sqlite package

* CGO_ENABLED=0 in unit tests

* disable for enterprise, too

* add driver name constant

* remove unused constants

* make test an integration one

* try integration tests without cgo

* implement error codes for modernc sqlite driver

* typo fix

* missing return

* use error pointer as an interface

* alias the driver

* update workspace, check for test errors too

* check error properly

* add missing driver after rebase

* fix missing import after rebase

* debugging, lets try again

* properly parse options, revert many previous changes

* remove another log

* better url parsing

* revert test rename, leave it for later

* revert reusedSession in unistore

* revert more code

* remove driver name

* revert formatting

* add integration test without cgo for sqlite

* remove tracing and logging

* bring driver alias back

* fix type

* wrong package
2025-09-02 17:24:30 +02:00
Bruno
f8cd7049e8 Secrets: garbage collection (#110247)
* clean up older secret versions

* start gargbage collection worker as background service

* make gen-go

* fix typo

* make update-workspace

* undo go mod changes

* undo go work sum changes

* Update pkg/registry/apis/secret/garbagecollectionworker/worker.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* Update pkg/registry/apis/secret/garbagecollectionworker/worker.go

Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>

* default gc_worker_batch_size to 1 minute

* fix typo

* fix typo

* add test to ensure cleaning up secure values is idempotent

* make gen-go

* make update-workspace

* undo go.mod and .sum changes

* undo enterprise imports

---------

Co-authored-by: Matheus Macabu <macabu.matheus@gmail.com>
Co-authored-by: Matheus Macabu <macabu@users.noreply.github.com>
2025-09-02 11:11:01 -03:00
Jean-Philippe Quéméner
6d5fe47790 fix(unified-storage): use contextual logger for permissions (#110455) 2025-09-02 15:53:37 +02:00
Jean-Philippe Quéméner
5fb72d1b04 fix(unified-strorage): optimize allocations (#110448) 2025-09-02 15:43:48 +02:00
Matheus Macabu
1e926a29c0 Secrets: Extract external facing decrypt types to apps (#110432) 2025-09-02 10:30:29 +02:00
maicon
fccf58add7 Unistore: Only Shadow Search Traffic when running on modes > 0 (#110302)
* Unistore: Only Shadow Search Traffic when running on modes > 0

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-01 16:14:53 -03:00
Peter Štibraný
4475e2ad19 getClientToDistributeRequest: remove logging of selected client, fix logging of error (#110403) 2025-09-01 10:57:25 +00:00
Peter Štibraný
da43e2ae07 Don't use transaction in ListModifiedSince. (#110392)
* Don't use transaction in ListModifiedSince.

To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.

* Fix integration test by converting SQL comments into template comments.
2025-09-01 11:39:02 +02:00
Todd Treece
9416abc146 Storage: Set default list limit to 500 (#110356) 2025-09-01 11:09:40 +02:00
Peter Štibraný
238f121e10 LastModifiedSince: return latestRV even when it hasn't changed. (#110391)
Return latestRV.
2025-09-01 08:45:40 +00:00
Will Assis
18bc69f5c6 unified-storage: Bleve test cleanup (#110240)
* consolidate index build tests

* shut up logging in test
2025-08-29 15:35:31 +00:00
Peter Štibraný
3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
Peter Štibraný
ad571b50e9 Create context with deadline inside goroutine. (#110297) 2025-08-28 15:33:34 +00:00