Commit Graph
118 Commits
Author SHA1 Message Date
Peter Štibraný 584287dc61 Add support for LIST with filtering on selectable fields 2025-12-15 12:10:43 +01:00
Jean-Philippe Quéméner 1f5fd1c0da chore(unified-storage): align how we do tracing (#114998) 2025-12-09 14:53:53 +01:00
Mihai Doarna b2e1b257b3 IAM: Add search for teams in app platform (#113503)
* add legacy search (wip)

* fix search field name

* implement team search endpoint

* generate openapi spec

* generate endpoints for frontend

* minor fixes

* fix issues found while testing

* add more fields to search result

* add basic unit tests

* add more unit tests

* improve getColumns() func in legacy search

* configure search endpoint in team.cue

* add team search handler

* add the searchTeams endpoint to manifest.cue

* make gofmt

* update openapi spec

* generate frontend endpoints

* remove unused field

* move fields defiitions to separate builder

* fix legacy search

* fix unit tests

* fix unit test

* address feedback

* fix unit test

* update openapi specs

* yarn generate-apis

* add missing unit tests
2025-12-08 15:02:59 +02:00
Misi 93ec32dd6a IAM: Add teams/{id}/groups as a custom endpoint to Teams API (#114228)
* Add teams/{id}/groups as a custom endpoint

* TeamGroupsHandler OSS and Ent registration

* Update OpenAPI spec

* Add indexer tests for external group mapping

* Remove noopsearch

* Remove unnecessary interface declaration, fixes

* Chores

* fmt

* Rename constant

* Align the rest to the changes of main

* Update workspace

* Add missing file
2025-11-25 14:19:57 +01:00
Jean-Philippe Quéméner b57e6383e4 refactor(unified-storage): move builders in their own package (#114375) 2025-11-25 10:58:03 +01:00
Peter Štibraný def4c644ec Use infra/log instead of slog in pkg/storage/unified (#114292)
Use infra/log instead of slog. Use log.FromContext to add contextual info to log messages.
2025-11-21 14:38:09 +01:00
Peter Štibraný 42db6324c9 Collect and log search stats. (#114107)
* Collect and log search stats.

* Fix compilation problems.
2025-11-19 16:52:47 +01:00
Rafael Bortolon Paulovic 0fc0fb8db5 chore: move enable search to US configuration (#114070) 2025-11-18 17:32:25 +01:00
Peter Štibraný ec74a721ea Be more tolerant to invalid JSON when parsing dashboards for indexing (#114040)
* Tags can only be string array.

* Be more lenient when parsing dashboard. Parse what we can, don't error out easily.

* Check element before parsing, log unexpected types.
2025-11-18 14:01:57 +01:00
Mustafa Sencer Özcan 6eac95f860 fix: inject index min update interval into resource server (#113816) 2025-11-13 09:27:54 +00:00
Ryan McKinley d4d8b2562e Chore: Update gocloud.dev, removing opencensus (#113056) 2025-10-28 09:10:37 +00:00
Denis VodopianovandDave Henderson 81683d554d chore : Deprecating FeatureToggles.IsEnabledGlobally (#112885)
* add deprecation on featuremgmt.IsEnabledGlobally

* add nolint reason

* add reasonable deprecation message

* remove junk edits

* add more nolints

* addressing review comments

* Update pkg/services/featuremgmt/models.go

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>

---------

Co-authored-by: Dave Henderson <dave.henderson@grafana.com>
2025-10-24 12:02:53 -04:00
ad9d8098ef IAM: Add email, login field validation to User create/update API (#112391)
* wip

* wip

* wip

(cherry picked from commit 8cedf25892)

* Search seems to be working, the validation is still wip

* Use keyword.Name analyzer for Filterable fields

* Only string fields should be indexed with keyword analyzer

* Change search query for email and login fields to use term query
* Remove unnecessary Exact from the resource protobuf definitions

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* Add legacy search support to the API

* Tests for legacy search, validate and integration tests for user

* Lint

* Add snapshot tests to userDocumentBuilder

* Address CodeQL issues

* Improvements, handle Mode2, tests should pass

* Change default limit from 0 to 1 for requests

* Cleanup

* Add fixme

* Update pkg/registry/apis/iam/register.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

* Update pkg/registry/apis/iam/user/legacy_search.go

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-10-23 11:29:02 +02:00
Will Assis 13f44336f2 unified-storage: remove full ngram feature toggle (#111994)
* unified-storage: remove full ngram feature toggle
2025-10-22 10:11:40 -04:00
Peter Štibraný 3391163f9c Check that we see less calls than allowed maximum. (#112785)
Check that we see less calls than allowed maximum.
2025-10-22 08:39:19 +00:00
Peter Štibraný a3c4f5cb9f Fix flaky TestConcurrentIndexUpdateAndSearchWithIndexMinUpdateInterval test (#112419)
Collect different RVs from update, and check that difference between them is at least 0.9*minInterval.
2025-10-15 11:17:44 +00:00
Peter Štibraný d801b87db9 LastImportTime for resource. (#112153)
* LastImportTime for resource.

* Make StorageBackendImpl implement GetResourceLastImportTimes

* More missing implementations of GetResourceLastImportTimes

* Fix import.

* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.

* Implement GetResourceLastImportTimes by mockStorageBackend

* Bump test tolerance.

* Fix postgres query and timezone.

* Fix postgres query and timezone.

* Make linter happy.
2025-10-09 11:27:11 +02:00
Peter Štibraný 3c6768460c Remove unused index_max_batch_size option to bleve backend. (#112171)
* Remove unused index_max_batch_size option to bleve backend.

* Remove batchSize argument.
2025-10-08 17:04:38 +02:00
Peter Štibraný 2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Will Assis 9bde3267bb unified-storage: dont index -- Grafana -- datasource dashboard fields (#111774)
* unified-storage: dont index -- Grafana -- datasource dashboard fields
2025-10-01 11:32:56 -04:00
Ryan McKinley 2f2289f226 Chore: Update authlib (foder as top level argument) (#111800) 2025-10-01 14:40:28 +00:00
Will Assis 870131a4aa unified-storage: restrict search to title of documents (#111842)
* restrict search to title of documents
2025-10-01 08:06:26 -04:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
Will Assis 33ff6dbb9e unified-storage: add feature flag to use ngram for indexing (#111265)
* unified-storage: add feature flag to use ngram instead of edge-ngram for indexing
2025-09-24 10:36:50 +02:00
Peter ŠtibranýandStephanie Hingtgen f062e5a85f Stop background tasks and when stopping bleve backend. (#111249)
Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-23 18:04:49 +02:00
Alexander Zobnin 72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Peter Štibraný 48250a1605 Introduced options to ignore old indexes based on build time and build version. (#111168) 2025-09-17 10:59:59 +02:00
Will Assis 6b2fb782c8 fix (unified-storage): search returning empty results when query includes small term (#111140)
* fix search not returning anything when a word in the query has less than 3 characters

* fix test name

* fmt

* remove trimspace
2025-09-16 08:55:23 -04:00
Ryan McKinley 0a79b3bdc5 Chore: Upgrade k8s.io/api v0.34.1 and grafana-app-sdk v0.43.1 (#111009) 2025-09-16 13:35:20 +03:00
Peter Štibraný f77e99d965 Store build time and build version into index. (#111010) 2025-09-16 10:52:30 +02:00
Peter Štibraný c6c8c6e928 Remove feature toggles from bleve backend. (#110951)
* Remove feature toggles from search backend.

* Remove extra import.

* Remove unnecessary code.
2025-09-11 16:08:41 +02:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný bd529226a3 unified-storage search: return results matching all search terms (#110672)
* Extract common indexing code.

* Extract common search code.

* Match all terms in the search query.
2025-09-05 11:00:54 +00:00
Peter Štibraný 642d43ff49 search-after-write: Reuse index even if docs count doesn't match (#110527)
* search-after-write: Reuse index even if docs count doesn't match

* Revert unrelated change.
2025-09-03 15:35:46 +00:00
Georges Chaudy 1758fa8fbb unistore: use scorch indexing instead of upside_down (#110463)
* use scorchwith in memory

* comments

* refactor
2025-09-03 11:06:08 +02:00
Jean-Philippe Quéméner 6d5fe47790 fix(unified-storage): use contextual logger for permissions (#110455) 2025-09-02 15:53:37 +02:00
Jean-Philippe Quéméner 5fb72d1b04 fix(unified-strorage): optimize allocations (#110448) 2025-09-02 15:43:48 +02:00
Will Assis 18bc69f5c6 unified-storage: Bleve test cleanup (#110240)
* consolidate index build tests

* shut up logging in test
2025-08-29 15:35:31 +00:00
Peter Štibraný 3a3ba483b1 unified-storage: Skip query when ListModifiedSince cannot return anything. (#110338)
* Skip query when ListModifiedSince cannot return anything.

* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.

* Add test for ListModifiedSince with same RV.
2025-08-29 14:49:20 +02:00
Peter Štibraný 7a8010be0c search-after-write: improve observability (#110288)
* Improve tracing and observability around updating of index during search.
2025-08-28 16:02:47 +02:00
Will Assis 7921c7da23 unified-storage: reuse index if RV is set even if it is outdated (#110184)
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
Peter Štibraný 4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
Will Assis 34496e137c unified-storage: Search after write save rv to index (#109641)
* save rv to index after index is built
2025-08-19 09:41:35 -04:00
Peter Štibraný c508b01cc5 chore: Close bleve indexes created in temp dir. (#109799)
* Close indexes created in temp dir.
2025-08-18 16:27:57 +02:00
Peter Štibraný 9b0dc4614c Cleanup old indexes in the same goroutine as build of the index. (#109382)
* Cleanup old indexes in the same goroutine as build of the index.

* Add test.
2025-08-08 14:59:59 +02:00
Stephanie Hingtgen 68059344f8 Unified storage: Remove unifiedStorageSearchPermissionFiltering feature flag (#109216)
* Unified storage: Remove unifiedStorageSearchPermissionFiltering ff

* unit test
2025-08-06 02:04:32 -06:00
Peter Štibraný c28b2215e0 Improve indexing observability (#108901)
* Add details to tracing spans when creating index.

* Log reason for building index.

* Log reason for building index.

* Remove initialization of labels to avoid unnecessary metrics.

* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.

* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00
Peter Štibraný 41319f90bb search: Handle index build errors gracefully (#108862)
* Close new index if we fail to build it.

* Respect context cancellation in getOrCreateIndex.
2025-07-29 17:40:16 +02:00