* fix(unified): default index path to ephemeral storage mount path
Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>
* Revert "fix: use memory index if index file already open (#115720)"
This reverts commit dc4c106e91.
* fix(unified): set index_path for tests
* chore(unified): re-add bolt open timeout and test for error handling
* chore(unified): return err on timeout
* chore(unified): revert changes to default, use DataPath if index_path not set
* chore(unified): add defaults.ini entry for unified_storage
This is needed to override using env. vars
* chore: revert unrelated diff
* chore: address code review comments
- reduce bolt timeout to 1s
- remove errIndexLocked err type
- add more information about index_path in defaults.ini
---------
Signed-off-by: Rafael Paulovic <rafael.paulovic@grafana.com>
* feat: add lock structure into bleve index files
* fix: another approach
* fix: new check
* fix: build in memory if index file already open
* fix: update workspace
* fix: add test
* refactor: update func signature
* fix: address comments
* fix: make const
* LastImportTime for resource.
* Make StorageBackendImpl implement GetResourceLastImportTimes
* More missing implementations of GetResourceLastImportTimes
* Fix import.
* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.
* Implement GetResourceLastImportTimes by mockStorageBackend
* Bump test tolerance.
* Fix postgres query and timezone.
* Fix postgres query and timezone.
* Make linter happy.
* Don't update index more often than specified index_min_update_interval.
* Add artificial sleep at the end of write operations.
* Improve test: check for number of update calls, make diff check less flaky.
* Make test less flaky by allowing for higher diff variance.
* Make test less flaky by allowing for expected update calls variance.
* Modify index eviction mechanism such that unowned indexes are also evicted.
* Propagate OwnsIndex function to bleve backend
Fix tests.
Stop eviction goroutine when stopping backend.
Make linter happy.
Make sure we stop backend created by tests.
Review suggestion.
Removed newline.
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.
* Remove reference to deprecated feature toggle.
* Remove feature toggle completely.
* Remove code related to indexing on watch events.
* Fix compilation error.
* Remove unused field.
* Add details to tracing spans when creating index.
* Log reason for building index.
* Log reason for building index.
* Remove initialization of labels to avoid unnecessary metrics.
* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.
* Revert removal of labels initialization.
* Get ResourceStats before indexing
* Replaced localcache.CacheService to handle expiration faster (localcache.CacheService / gocache.Cache only expires values at specific interval, but we need to close index faster)
* singleflight getOrBuildIndex for the same key
* expire only in-memory indexes
* file-based indexes have new name on each rebuild
* Sanitize file path segments, verify that generated path is within the root dir.
* Add comment and test for cleanOldIndexes.
* Fix CodeQL warnings
* Also validate if path is a file above the safe dir
* Lint
* [REVIEW] reduce possibility of exploits
* Comment
* Remove test scenario
* refactor grafana_index_server_index_size to calculate in a goroutine instead of at scrape time and remove grafana_index_server_indexed_docs metric
* use wire to inject bleve index metrics
* remove sprinkles metrics from bleve index metrics
* log error when trying to calculate file index size and bump interval to 1m instead of 5s
* wires up dashboards page to be able to sort by usage stats (sprinkles)
* dont mutate field
* use better type for field
* adds tests. Had to export some types and put the field type back to object.
* frontend asks for sort field in response if needed
* adds some unit tests for getSortOptions
* use Record instead of object
* prettier
* adds ternaries, another unit test
* add feature flag
* puts search permission filtering behind a feature flag
* fixes issue with doc match id. When the match is for an in-memory index, the internal id is a string (this is what we expected). However, when its a file-based index, the internal id is a binary encoded int64 that point to something internally. So to get the id, we need to use ExternalID() instead of relying on the indexInternalID to be the correct format.
* adds debug log
* update comment
* formatting
* fix bug when parsing results in search handler
* applies permissions filtering to bleve query
* formatting
* wraps in check for access being present, adds some comments
* update go mod
* fix tests
* add dep owner
* fix go mod
* add space after //
* clean up returns
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
* fixed formatting
* Uses single checker since index is for single resource. Passes folderId using dvReader to checker func. Adds debug logging.
* handles federation with index permission checkers
* formatting
* move import
---------
Co-authored-by: Bruno Abrantes <bruno.abrantes@grafana.com>
Rafael Bortolon Paulovic
Ryan McKinley
Mustafa Sencer Özcan
Jean-Philippe Quéméner
Peter Štibraný
Will Assis
Alexander Zobnin
Georges Chaudy
Leonor Oliveira
Karl Persson
Scott Lepper
owensmallwood
