Commit Graph
24 Commits
Author SHA1 Message Date
Rafael Bortolon Paulovic 987c1fc6b6 feat(unified): add index scoring model config (#116210)
* feat(unified): add bm25 index scoring model

We want try BM25 scoring model since they have global scoring which we can probably re-use for fan-in/fan-out logic

https://github.com/blevesearch/bleve/blob/32d98823c4b7482c62cc6c847508ed7659c23c37/docs/scoring.md#global-scoring

* fix(plugins): update plugin test data
2026-01-14 12:07:53 +01:00
Rafael Bortolon Paulovic 0d7f46c08a chore(unified): remove unifiedStorageSearch feature toggle (#116109) 2026-01-12 13:22:48 +01:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Rafael Bortolon Paulovic 0fc0fb8db5 chore: move enable search to US configuration (#114070) 2025-11-18 17:32:25 +01:00
Mustafa Sencer Özcan 6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Denis VodopianovandDave Henderson 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis 33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter Štibraný 48250a1605 Introduced options to ignore old indexes based on build time and build version. (#111168) 2025-09-17 10:59:59 +02:00
Peter Štibraný f77e99d965 Store build time and build version into index. (#111010) 2025-09-16 10:52:30 +02:00
Peter Štibraný c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný 4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
Arati R. 0982cfd9a0 Unified Storage/Search: Add max count config for indexing (#107255)
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold

* Refactor mockResourceIndex

* Test setting of indexing threshold configs

* Tweak comments, log

* Fix logging in buildEmptyIndex

* Export and reuse TestDocumentBuilderSupplier

* Reuse MockResourceIndex
2025-06-27 14:00:39 +02:00
Mustafa Sencer Özcan b48337a7c8 feat: introduce ttl cache for bleve indices (#106842) 2025-06-17 16:28:51 +02:00
Stephanie Hingtgen 5135d5c87d Unified storage: Reconstruct index in the background every 24h (#106422) 2025-06-12 14:34:48 -05:00
Leonor Oliveira e9ed7223a6 Use authlib repo. Use otel (#103178)
* Use authlib repo. Use otel

* Use interceptors on the provider level

* Create a new wire set with otel

* Lint

* Fix test

* make update-workflow

* make update-workspace

* make update-workspace. Try to add authlib as enterprise imports

* make update-workspace
2025-04-07 15:47:40 +02:00
Ryan McKinley 6781612335 Provisioning: Enable search when provisioning is enabled (#103208) 2025-04-02 11:20:28 +03:00
Will Assis 31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
Peter ŠtibranýandGeorges Chaudy d0394bfa7e Extract NewSearchOptions from unified sql client setup. (#100719)
* Extract NewSearchOptions from unified sql client setup.

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-02-14 16:39:48 +01:00