Commit Graph
24 Commits
Author SHA1 Message Date
Rafael Bortolon Paulovic 987c1fc6b6 feat(unified): add index scoring model config (#116210)
* feat(unified): add bm25 index scoring model

We want try BM25 scoring model since they have global scoring which we can probably re-use for fan-in/fan-out logic

https://github.com/blevesearch/bleve/blob/32d98823c4b7482c62cc6c847508ed7659c23c37/docs/scoring.md#global-scoring

* fix(plugins): update plugin test data
2026-01-14 12:07:53 +01:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Jean-Philippe Quéméner b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Will Assis 33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter ŠtibranýandStephanie Hingtgen f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
Will Assis 6b2fb782c8 fix (unified-storage): search returning empty results when query includes small term (#111140)
* fix search not returning anything when a word in the query has less than 3 characters

* fix test name

* fmt

* remove trimspace
2025-09-16 08:55:23 -04:00
Peter Štibraný c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný bd529226a3 unified-storage search: return results matching all search terms (#110672)
* Extract common indexing code.

* Extract common search code.

* Match all terms in the search query.
2025-09-05 11:00:54 +00:00
Will Assis 7921c7da23 unified-storage: reuse index if RV is set even if it is outdated (#110184)
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
Peter Štibraný 4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
Peter Štibraný c508b01cc5 chore: Close bleve indexes created in temp dir. (#109799)
* Close indexes created in temp dir.
2025-08-18 16:27:57 +02:00
Stephanie Hingtgen 68059344f8 Unified storage: Remove unifiedStorageSearchPermissionFiltering feature flag (#109216)
* Unified storage: Remove unifiedStorageSearchPermissionFiltering ff

* unit test
2025-08-06 02:04:32 -06:00
Peter Štibraný c28b2215e0 Improve indexing observability (#108901)
* Add details to tracing spans when creating index.

* Log reason for building index.

* Log reason for building index.

* Remove initialization of labels to avoid unnecessary metrics.

* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.

* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00
Jean-Philippe Quéméner 002f46736a refactor(unified-storage): move generated protos to own pkg (#105356) 2025-05-15 21:36:52 +02:00
Georges Chaudy 15b3de5893 search: add index batching (#104163)
* add basic search backend integration tests

* add search backend benchmark

* add benchmark indexServer

* fix

* lint

* add more tests

* lint

* do not use the poller

* batch write

* refactor and add tests

* improvements

* improvements

* cleanup

* only observe index success

* add monitorIndexEvents method

* nit use switch instead of if

* make newIndexQueueProcessor private

* simplify runProcessor

* go lint
2025-05-09 15:36:21 +02:00
Mariell Hoversholm e1e1d3fd9f Fix: Prints should always include new lines (#102795)
* CI: Allow Bench conversion to fail

We shouldn't mark PRs and commits as X if they fail to convert logs with Bench.

* Fix: Prints should always include new lines

* fix: remove unused import
2025-03-27 12:27:53 +01:00
Scott Lepper 2aae523c3f Search: Fix title filter overmatching (#102547)
* fix issues with over matching
* search wildcard for backward compatibility
2025-03-21 09:57:20 -04:00
Will Assis 31a371e385 fix(unified-storage): remove bleve index metric from global scope (#101825)
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric

* use wire to inject bleve index metrics

* remove sprinkles metrics from bleve index metrics

* log error when trying to calculate file index size and bump interval to 1m instead of 5s
2025-03-13 10:09:38 -04:00
Ryan McKinley 64f1df69d5 Search: Rename managed object index (#101856) 2025-03-10 19:48:53 +03:00
owensmallwood 827da46c51 Unified Storage: Remove wildcard queries (#101441)
* WIP adding custom analyzer so we can do substring search efficiently

* Adding unit tests for title search

* formatting

* adds more title search unit tests

* organize helpers

* fixes issue caused by having two title mappings

* Removes camelcase token filter since it prevents you from searching for a substring of chars and numbers. Adds regression test.

* adds back mapping for title_phrase

* use simple analyzer for input query string so it doesn't filter out english stop words

* ran bleve tests, table snapshots updated

* ignore linter for "unused" test functions. They are very helpful for troubleshooting search. Keeping them.

* only log total hits and query cost if result not nil

* fixes failing test - one more field because there are two title mappings now

* fix test

* fixes test - only take first item when its the title

* Adds separate internal field for title ngram mapping.

When searching with a query, results are sorted by score desc.

When searching without a query, results are sorted by title desc.

Adjusts ngram max to be 10.

Text queries are a disjunction of an exact match, phrase match, and a match. Boosted to have priority in that order.

Adds more unit tests for searching.

* linter

* fix test

* ran tests - generated new test dash json

* sort by title phrase instead of title

* fix test - not relying on /apis/dashboard/search to apply title sorting anymore
2025-03-07 09:51:15 -06:00