Commit Graph
89 Commits
Author SHA1 Message Date
Ryan McKinley 48625d67e5 Chore: update blevesearch dependencies (#116251) 2026-01-14 12:15:19 +00:00
Ryan McKinley 040854c8af Search: Allow query field selection (#116238) 2026-01-14 11:55:05 +00:00
Rafael Bortolon Paulovic 987c1fc6b6 feat(unified): add index scoring model config (#116210)
* feat(unified): add bm25 index scoring model

We want try BM25 scoring model since they have global scoring which we can probably re-use for fan-in/fan-out logic

https://github.com/blevesearch/bleve/blob/32d98823c4b7482c62cc6c847508ed7659c23c37/docs/scoring.md#global-scoring

* fix(plugins): update plugin test data
2026-01-14 12:07:53 +01:00
Ryan McKinley 1b52718c23 Search: include panel titles and types in index (#115742) 2026-01-12 13:21:03 +01:00
Rafael Bortolon Paulovic 1c5caeb987 fix(unified): err on timeout to open index (#115953)
* fix(unified): default index path to ephemeral storage mount path

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>

* Revert "fix: use memory index if index file already open (#115720)"

This reverts commit dc4c106e91.

* fix(unified): set index_path for tests

* chore(unified): re-add bolt open timeout and test for error handling

* chore(unified): return err on timeout

* chore(unified): revert changes to default, use DataPath if index_path not set

* chore(unified): add defaults.ini entry for unified_storage

This is needed to override using env. vars

* chore: revert unrelated diff

* chore: address code review comments

- reduce bolt timeout to 1s
- remove errIndexLocked err type
- add more information about index_path in defaults.ini

---------

Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>
2026-01-09 13:48:54 +01:00
Mustafa Sencer Özcan dc4c106e91 fix: use memory index if index file already open (#115720)
* feat: add lock structure into bleve index files

* fix: another approach

* fix: new check

* fix: build in memory if index file already open

* fix: update workspace

* fix: add test

* refactor: update func signature

* fix: address comments

* fix: make const
2026-01-02 13:51:51 +01:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Jean-Philippe Quéméner b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Peter Štibraný 2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Will Assis 870131a4aa unified-storage: restrict search to title of documents (#111842)
* restrict search to title of documents
2025-10-01 08:06:26 -04:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis 33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter ŠtibranýandStephanie Hingtgen f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
Alexander Zobnin 72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Peter Štibraný 48250a1605 Introduced options to ignore old indexes based on build time and build version. (#111168) 2025-09-17 10:59:59 +02:00
Will Assis 6b2fb782c8 fix (unified-storage): search returning empty results when query includes small term (#111140)
* fix search not returning anything when a word in the query has less than 3 characters

* fix test name

* fmt

* remove trimspace
2025-09-16 08:55:23 -04:00
Peter Štibraný f77e99d965 Store build time and build version into index. (#111010) 2025-09-16 10:52:30 +02:00
Peter Štibraný c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný bd529226a3 unified-storage search: return results matching all search terms (#110672)
* Extract common indexing code.

* Extract common search code.

* Match all terms in the search query.
2025-09-05 11:00:54 +00:00
Peter Štibraný 642d43ff49 search-after-write: Reuse index even if docs count doesn't match (#110527)
* search-after-write: Reuse index even if docs count doesn't match

* Revert unrelated change.
2025-09-03 15:35:46 +00:00
Georges Chaudy 1758fa8fbb unistore: use scorch indexing instead of upside_down (#110463)
* use scorchwith in memory

* comments

* refactor
2025-09-03 11:06:08 +02:00
Jean-Philippe Quéméner 6d5fe47790 fix(unified-storage): use contextual logger for permissions (#110455) 2025-09-02 15:53:37 +02:00
Jean-Philippe Quéméner 5fb72d1b04 fix(unified-strorage): optimize allocations (#110448) 2025-09-02 15:43:48 +02:00
Will Assis 18bc69f5c6 unified-storage: Bleve test cleanup (#110240)
* consolidate index build tests

* shut up logging in test
2025-08-29 15:35:31 +00:00
Peter Štibraný 3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
Peter Štibraný 7a8010be0c search-after-write: improve observability (#110288)
* Improve tracing and observability around updating of index during search.
2025-08-28 16:02:47 +02:00
Will Assis 7921c7da23 unified-storage: reuse index if RV is set even if it is outdated (#110184)
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
Peter Štibraný 4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
Will Assis 34496e137c unified-storage: Search after write save rv to index (#109641)
* save rv to index after index is built
2025-08-19 09:41:35 -04:00
Peter Štibraný c508b01cc5 chore: Close bleve indexes created in temp dir. (#109799)
* Close indexes created in temp dir.
2025-08-18 16:27:57 +02:00
Peter Štibraný 9b0dc4614c Cleanup old indexes in the same goroutine as build of the index. (#109382)
* Cleanup old indexes in the same goroutine as build of the index.

* Add test.
2025-08-08 14:59:59 +02:00
Stephanie Hingtgen 68059344f8 Unified storage: Remove unifiedStorageSearchPermissionFiltering feature flag (#109216)
* Unified storage: Remove unifiedStorageSearchPermissionFiltering ff

* unit test
2025-08-06 02:04:32 -06:00
Peter Štibraný c28b2215e0 Improve indexing observability (#108901)
* Add details to tracing spans when creating index.

* Log reason for building index.

* Log reason for building index.

* Remove initialization of labels to avoid unnecessary metrics.

* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.

* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00
Peter Štibraný 41319f90bb search: Handle index build errors gracefully (#108862)
* Close new index if we fail to build it.

* Respect context cancellation in getOrCreateIndex.
2025-07-29 17:40:16 +02:00
Peter Štibraný d9daf2e424 search: Track number of open indexes per storage type. (#108842)
* Track number of open indexes per storage type.

* Fix tests after changing description.
2025-07-29 12:46:34 +02:00
Peter Štibraný 8fd5739576 [unified-storage/search] Don't expire file-based indexes, check for resource stats when building index on-demand (#107886)
* Get ResourceStats before indexing
* Replaced localcache.CacheService to handle expiration faster (localcache.CacheService / gocache.Cache only expires values at specific interval, but we need to close index faster)
* singleflight getOrBuildIndex for the same key
* expire only in-memory indexes
* file-based indexes have new name on each rebuild
* Sanitize file path segments, verify that generated path is within the root dir.
* Add comment and test for cleanOldIndexes.
2025-07-10 11:54:10 +00:00
Georges Chaudy 5150ebfba8 unistore: index search references (#106954) 2025-06-24 14:54:07 -05:00
Mustafa Sencer Özcan b48337a7c8 feat: introduce ttl cache for bleve indices (#106842) 2025-06-17 16:28:51 +02:00
Jean-Philippe Quéméner 002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Georges Chaudy e162c69c34 search: fix document missing at startup (#105198)
* fix document missing at startup

* go-lint

* fix tests

* fix tests

* fix integration tests now that we are storing real values
2025-05-12 07:56:25 +02:00
Georges Chaudy 15b3de5893 search: add index batching (#104163)
* add basic search backend integration tests

* add search backend benchmark

* add benchmark indexServer

* fix

* lint

* add more tests

* lint

* do not use the poller

* batch write

* refactor and add tests

* improvements

* improvements

* cleanup

* only observe index success

* add monitorIndexEvents method

* nit use switch instead of if

* make newIndexQueueProcessor private

* simplify runProcessor

* go lint
2025-05-09 15:36:21 +02:00
Will Assis db1f1c5df9 search: add manager kind to bleve (#103305) 2025-04-03 10:02:02 +03:00