Commit Graph

394 Commits

Author SHA1 Message Date
Peter Štibraný
357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
maicon
b3ed600bca Revert "Unistore: Add validation for resource names" (#111408)
Revert "Unistore: Add validation for resource names (#110990)"

This reverts commit b63e3fd3ae.
2025-09-19 18:03:23 +00:00
maicon
b63e3fd3ae Unistore: Add validation for resource names (#110990)
* Unistore: Add validation for resource names

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-19 09:37:17 -03:00
Alexander Zobnin
72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Ryan McKinley
0a79b3bdc5 Chore: Upgrade k8s.io/api v0.34.1 and grafana-app-sdk v0.43.1 (#111009) 2025-09-16 13:35:20 +03:00
Georges Chaudy
c251ebf4d5 kvstore: merge the metadata store into the datastore (#110334)
* migrate eventstore to datastore

* Add folder to event key

* lint

* lint

* lint

* lint

* remove foundkye

* refactor the Keys methods to move the Sort outside the ListKey method

* remove bad import

* fix missing params

* lint

* fix test

* perf improvement
2025-09-16 12:16:29 +02:00
Alexander Zobnin
294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
owensmallwood
7ce971cba1 Unified Storage: Adds pruner for kv eventstore (#110785)
* Adds pruner for eventstore - default 24 hours. Adds tests.

* update comment

* remove delay on startup. formatting

* updates log message type and removes useless comment

* caller handles goroutine for runCleanupOldEvents()

* simplify timestamp extraction

* adds config for event pruning interval

* uses start and end key to get all expired events

* remove sort when listing keys in event pruner - order doesnt matter

* use snowflake constants

* log when we delete 0 rows

* pass time.Time to cleanup old events func
2025-09-12 14:40:16 -06:00
owensmallwood
310893292f Unified Storage: Add sort order to keys func in datastore (#110714)
* Add sort order to keys func in datastore. Add test to not prune deleted events.

* include sort field in the ListRequestKey instead of it being a separate param
2025-09-11 14:12:30 -06:00
Peter Štibraný
6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný
b3cc317cbc unified-storage search: Return bad request error for incomplete request. (#110802)
Return bad request error for incomplete request.
2025-09-09 14:15:45 +02:00
Will Assis
005da25698 fix: ListManagedObjects and CountManagedObjects panic when search is not configured (#110726)
fix ListManagedObjects and CountManagedObjects panicking when search index not configured
2025-09-05 20:58:12 +00:00
owensmallwood
dc1c5a610c Unified Storage: Fix broken tests (#110710)
fix broken tests
2025-09-05 17:19:13 +00:00
owensmallwood
d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Ryan McKinley
eeb940e733 Chore: Replace hand crafted mocks with mockery (#110627) 2025-09-05 07:13:15 +00:00
Will Assis
ea7c370edd unified-storage: add ListSinceModified to kv store (#110250)
* implement ListKeysSince in event store

 implement data store version

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-09-04 12:26:40 -04:00
maicon
fa20755d4b fix: Synchronously notify subscribers when writing Create events to CDK backend (#110476)
* Remove goroutine when writing events on CDK backend

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-03 11:49:04 -03:00
maicon
fccf58add7 Unistore: Only Shadow Search Traffic when running on modes > 0 (#110302)
* Unistore: Only Shadow Search Traffic when running on modes > 0

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-01 16:14:53 -03:00
Peter Štibraný
4475e2ad19 getClientToDistributeRequest: remove logging of selected client, fix logging of error (#110403) 2025-09-01 10:57:25 +00:00
Todd Treece
9416abc146 Storage: Set default list limit to 500 (#110356) 2025-09-01 11:09:40 +02:00
Peter Štibraný
ad571b50e9 Create context with deadline inside goroutine. (#110297) 2025-08-28 15:33:34 +00:00
Peter Štibraný
7a8010be0c search-after-write: improve observability (#110288)
* Improve tracing and observability around updating of index during search.
2025-08-28 16:02:47 +02:00
Will Assis
7921c7da23 unified-storage: reuse index if RV is set even if it is outdated (#110184)
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
Mustafa Sencer Özcan
3056924d10 fix: enforce timeout for requests run in qos (#110162) 2025-08-26 18:40:22 +02:00
Matheus Macabu
78ac555c4d CI: Run gofmt on Linting action (#110159)
* CI: Run gofmt on Linting action

* Chore: Run gofmt on all packages
2025-08-26 16:14:03 +02:00
Peter Štibraný
067b6e20a2 Fix name of the summary, remove _total suffix (#110143) 2025-08-26 08:48:05 +00:00
Peter Štibraný
4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
owensmallwood
cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Will Assis
34496e137c unified-storage: Search after write save rv to index (#109641)
* save rv to index after index is built
2025-08-19 09:41:35 -04:00
Will Assis
43cda1f1a4 unified-storage: disable watcher when search-after-write feature flag is enabled (#109634)
* disable indexer watcher when search-after-write feature flag is enabled
2025-08-14 10:18:09 -04:00
Ryan McKinley
d3df5b8ddd Secrets: Manage secure values inside any resource (#107803) 2025-08-14 12:31:24 +00:00
Mustafa Sencer Özcan
7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Peter Štibraný
298b492f1a search queue index mutex (#109474)
Use custom mutex for index, and don't hold it during BulkIndex.
2025-08-11 16:20:47 +00:00
Ryan McKinley
e0404f924c K8s/SecureValues: Wire InlineSecureValueSupport to apistore (#109449)
* inline wire

* extra fields

* add variable

* wire
2025-08-11 15:22:56 +03:00
owensmallwood
c07c0f27d2 Unified Storage: Makes writing and reading index thread-safe in the queue processor (#109420)
makes writing and reading index thread-safe
2025-08-08 12:53:01 -06:00
owensmallwood
e667dfed29 Unified Storage: Update index when indexqueueprocessor exists (#109351)
* Update readme

* When queue processor already exists, update the index on it in case it has been closed.

* Adds regression test.
2025-08-07 11:38:24 -06:00
Bruno Abrantes
f3b8a891af chore: use native histogram for search shadow traffic match percentage (#109189)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-08-06 08:56:59 +02:00
Bruno Abrantes
cb921dc47a feat: compare legacy and unified search results via histogram (#109022)
* feat: compare legacy and unified search results via histogram

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: handle cases where request type is not set

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: use struct instead of bool because it's more memory efficient

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: calculate recall percentage rather than union between legacy and unified

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-08-05 11:17:32 +02:00
Peter Štibraný
372f115db9 Finish building index even if original context is canceled. (#108969) 2025-07-31 13:45:55 +02:00
Peter Štibraný
c28b2215e0 Improve indexing observability (#108901)
* Add details to tracing spans when creating index.

* Log reason for building index.

* Log reason for building index.

* Remove initialization of labels to avoid unnecessary metrics.

* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.

* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00
Peter Štibraný
41319f90bb search: Handle index build errors gracefully (#108862)
* Close new index if we fail to build it.

* Respect context cancellation in getOrCreateIndex.
2025-07-29 17:40:16 +02:00
Peter Štibraný
d9daf2e424 search: Track number of open indexes per storage type. (#108842)
* Track number of open indexes per storage type.

* Fix tests after changing description.
2025-07-29 12:46:34 +02:00
Will Assis
aa7ae5fc65 unified-storage: add tracing to distributor methods (#108791)
* add tracing to distributor methods
2025-07-28 12:35:20 -04:00
Will Assis
f67713726b add debug logs to search server distributor (#108679) 2025-07-25 12:36:35 +00:00
Arati R.
16aee8047c Unified Search: Randomise instance selected by distributor (#108619)
* Randomise instance selected by distributor

* add namespace to distributor logging

---------

Co-authored-by: Will Assis <william@williamassis.com>
2025-07-24 19:38:35 +00:00
Will Assis
981fdb29d4 update storage-api to only build index if it owns the namespace (#108418)
* update storage-api to only build index if it owns the namespace

---------

Co-authored-by: Mustafa Sencer Özcan <mustafasencer.ozcan@grafana.com>
2025-07-23 15:59:24 -04:00
Will Assis
34f1be796d unified-storage: expose ring replication factor config (#106345)
* config ring replication factor

* change default

* rename

* fix test

* fix
2025-07-22 09:32:22 +02:00
Bruno Abrantes
6c84461b7a fix: Always return searchWrapper to enable shadow traffic (#108373)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-21 13:53:44 +02:00
maicon
4b440cf45d Remove timeout info from logs when shadow request to search succeeds (#108324)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-07-18 14:03:42 +00:00
maicon
2dba473015 Feature/unified storage search dual reader (#108291)
* Add UnifiedStorageSearchDualReaderEnabled feature flag

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Refactor UniSearch Dual Reader

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* Run make gen-feature-toggles

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

* fix: unit tests search_client

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* feat: cancels shadow search requests after 500ms

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
Co-authored-by: Will Assis <william@williamassis.com>
Co-authored-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-07-18 14:43:56 +02:00