Commit Graph
404 Commits
Author SHA1 Message Date
maicon 2c5ccd3283 Unistore: ignore name when validating collection keys (#112086)
Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-10-07 06:45:59 +00:00
Peter Štibraný 2d801eed3c Remove unused arguments from ResourceIndex and SearchBackend interfaces. (#112043)
* Remove unused arguments from ResourceIndex and SearchBackend interfaces.
2025-10-06 11:47:52 +02:00
Peter Štibraný a44af81082 Unified storage search: Introduce min index update interval (#111978)
* Don't update index more often than specified index_min_update_interval.

* Add artificial sleep at the end of write operations.

* Improve test: check for number of update calls, make diff check less flaky.

* Make test less flaky by allowing for higher diff variance.

* Make test less flaky by allowing for expected update calls variance.
2025-10-06 10:02:03 +02:00
Ryan McKinley c530cacb1c Storage: Propagate RV to server on update+delete (#111866) 2025-10-01 18:00:02 +03:00
Ryan McKinley 2f2289f226 Chore: Update authlib (foder as top level argument) (#111800) 2025-10-01 14:40:28 +00:00
Peter Štibraný 707c486a46 Rebuild search indexes asynchronously (#111829)
* Add "debouncer" queue, which can combine incoming elements.

* Rebuild indexes asynchronously.

* Remove duplicate method.

* Fix bleve tests.

* Extracted combineRebuildRequests and added test for it.

* Add TestShouldRebuildIndex

* Added TestFindIndexesForRebuild

* Added TestFindIndexesForRebuild

* Introduce index_rebuild_workers option.

* Add metric for rebuild queue length.

* Add TestRebuildIndexes.

* Fix import.

* Linter, review feedback.
2025-10-01 11:52:09 +02:00
Ryan McKinley cfbf64c3fd Validation: Move validation into apimachinery package (#111736) 2025-09-30 12:59:33 +00:00
Stephanie Hingtgen e01e61a156 Trash: Keep provisioned objects out of the trash (#111714) 2025-09-28 20:33:57 +00:00
maicon 2e734549b5 Revert "Revert "Unistore: Add validation for resource names"" (#111648)
* Revert "Revert "Unistore: Add validation for resource names" (#111408)"

This reverts commit b3ed600bca.


---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-28 20:53:07 +03:00
Ryan McKinley bdefea9acf Storage: Fast path when list with metadata.name=XYZ (#111630) 2025-09-25 15:54:38 +00:00
Peter Štibraný 357aa7d314 Improve search index eviction (#111542)
* Modify index eviction mechanism such that unowned indexes are also evicted.

* Propagate OwnsIndex function to bleve backend

Fix tests.

Stop eviction goroutine when stopping backend.

Make linter happy.

Make sure we stop backend created by tests.

Review suggestion.

Removed newline.
2025-09-24 16:54:35 +02:00
maicon b3ed600bca Revert "Unistore: Add validation for resource names" (#111408)
Revert "Unistore: Add validation for resource names (#110990)"

This reverts commit b63e3fd3ae.
2025-09-19 18:03:23 +00:00
maicon b63e3fd3ae Unistore: Add validation for resource names (#110990)
* Unistore: Add validation for resource names

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-19 09:37:17 -03:00
Alexander Zobnin 72d212c5f9 Authlib: Update authz client to use zookies (#111291)
* Authlib: Update authz client to use zookies

* fix zookie return

* fix linter
2025-09-18 16:24:22 +02:00
Ryan McKinley 0a79b3bdc5 Chore: Upgrade k8s.io/api v0.34.1 and grafana-app-sdk v0.43.1 (#111009) 2025-09-16 13:35:20 +03:00
Georges Chaudy c251ebf4d5 kvstore: merge the metadata store into the datastore (#110334)
* migrate eventstore to datastore

* Add folder to event key

* lint

* lint

* lint

* lint

* remove foundkye

* refactor the Keys methods to move the Sort outside the ListKey method

* remove bad import

* fix missing params

* lint

* fix test

* perf improvement
2025-09-16 12:16:29 +02:00
Alexander ZobninandJo Garnier 294fd943c0 Chore: Update authlib (#110880)
* Chore: Update authlib

* exclude incompatible version of github.com/grafana/gomemcache

* Update go-jose to v4

* fix jose imports

* remove jose v3 from go.mod

* fix tests

* fix serialize

* fix failing live tests

* add v1 of ES256 testkeys. Port tests to use ES256 instead of HS256

* accept more signature algs for okta and azuread

* azure social graph token sig

* accept more signature algs for oauth refresh and jwt auth

* update workspace

* add a static signer for inproc

* rebase and fix ext_jwt

* fix jwt tests

* apply alex patch on gomemcache

* update linting

* fix ext_jwt panic

* update workspaces

---------

Co-authored-by: Jo Garnier <git@jguer.space>
2025-09-15 12:45:15 +02:00
owensmallwood 7ce971cba1 Unified Storage: Adds pruner for kv eventstore (#110785)
* Adds pruner for eventstore - default 24 hours. Adds tests.

* update comment

* remove delay on startup. formatting

* updates log message type and removes useless comment

* caller handles goroutine for runCleanupOldEvents()

* simplify timestamp extraction

* adds config for event pruning interval

* uses start and end key to get all expired events

* remove sort when listing keys in event pruner - order doesnt matter

* use snowflake constants

* log when we delete 0 rows

* pass time.Time to cleanup old events func
2025-09-12 14:40:16 -06:00
owensmallwood 310893292f Unified Storage: Add sort order to keys func in datastore (#110714)
* Add sort order to keys func in datastore. Add test to not prune deleted events.

* include sort field in the ListRequestKey instead of it being a separate param
2025-09-11 14:12:30 -06:00
Peter Štibraný 6fa6a5708a Cleanup of old search functionality (#110861)
* Remove support for initMinSize.
Remove support for searchAfterWrite option, now it defaults to true.

* Remove reference to deprecated feature toggle.

* Remove feature toggle completely.

* Remove code related to indexing on watch events.

* Fix compilation error.

* Remove unused field.
2025-09-11 08:23:03 +00:00
Peter Štibraný b3cc317cbc unified-storage search: Return bad request error for incomplete request. (#110802)
Return bad request error for incomplete request.
2025-09-09 14:15:45 +02:00
Will Assis 005da25698 fix: ListManagedObjects and CountManagedObjects panic when search is not configured (#110726)
fix ListManagedObjects and CountManagedObjects panicking when search index not configured
2025-09-05 20:58:12 +00:00
owensmallwood dc1c5a610c Unified Storage: Fix broken tests (#110710)
fix broken tests
2025-09-05 17:19:13 +00:00
owensmallwood d715bda8af Unified Storage: Adds pruner to kv backend (#110549)
* WIP adding pruner to kv store impl

* pruner only keeps 20 most recent versions

* ignore grafana-kv-data folder

* extracts some stuff to pruner.go file. Adds tests. Adds kvBackendOptions.

* update logging, comments, exports kvbackendoptions fields

* update nooppruner ref

* fixes field casing in test

* fix test

* linter fixes

* remove comment

* make KvStorageBackend private

* Adds pruner key validation and tests. Fixes broken tests.

* update error message when validating pruner key
2025-09-05 10:02:11 -06:00
Ryan McKinley eeb940e733 Chore: Replace hand crafted mocks with mockery (#110627) 2025-09-05 07:13:15 +00:00
Will AssisandGeorges Chaudy ea7c370edd unified-storage: add ListSinceModified to kv store (#110250)
* implement ListKeysSince in event store

 implement data store version

---------

Co-authored-by: Georges Chaudy <chaudyg@gmail.com>
2025-09-04 12:26:40 -04:00
maicon fa20755d4b fix: Synchronously notify subscribers when writing Create events to CDK backend (#110476)
* Remove goroutine when writing events on CDK backend

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-03 11:49:04 -03:00
maicon fccf58add7 Unistore: Only Shadow Search Traffic when running on modes > 0 (#110302)
* Unistore: Only Shadow Search Traffic when running on modes > 0

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>

---------

Signed-off-by: Maicon Costa <maiconscosta@gmail.com>
2025-09-01 16:14:53 -03:00
Peter Štibraný 4475e2ad19 getClientToDistributeRequest: remove logging of selected client, fix logging of error (#110403) 2025-09-01 10:57:25 +00:00
Todd Treece 9416abc146 Storage: Set default list limit to 500 (#110356) 2025-09-01 11:09:40 +02:00
Peter Štibraný ad571b50e9 Create context with deadline inside goroutine. (#110297) 2025-08-28 15:33:34 +00:00
Peter Štibraný 7a8010be0c search-after-write: improve observability (#110288)
* Improve tracing and observability around updating of index during search.
2025-08-28 16:02:47 +02:00
Will Assis 7921c7da23 unified-storage: reuse index if RV is set even if it is outdated (#110184)
* unified-storage: reuse index if RV is set even if it is outdated and searchAfterWrite FF is enabled
2025-08-27 11:10:54 -04:00
Mustafa Sencer Özcan 3056924d10 fix: enforce timeout for requests run in qos (#110162) 2025-08-26 18:40:22 +02:00
Matheus Macabu 78ac555c4d CI: Run gofmt on Linting action (#110159)
* CI: Run gofmt on Linting action

* Chore: Run gofmt on all packages
2025-08-26 16:14:03 +02:00
Peter Štibraný 067b6e20a2 Fix name of the summary, remove _total suffix (#110143) 2025-08-26 08:48:05 +00:00
Peter Štibraný 4c996a8a7f search: Guarantee search-after-write consistency (#109972) 2025-08-25 10:13:07 +02:00
owensmallwood cace999671 Unified Storage: Add ListSinceModified to StorageBackend (#109697)
* WIP added ListSinceModified to StorageBackend interface

* fix compile time check

* Fix method name

* Fix naming

* fix the rest of the ListSinceModified names

* Uses resource key without name field

* get latest rv from resource_version. Update test.

* adds moar tests

* adds method stub for ListModifiedSince to other StorageBackend implementations

* adds dummy impl to noop storage backend for ListModifiedSince

* skip tests for badger kv backend for now

* fixes tests and adds badgerkv impl for ListModifiedSince

* add badger kv impl

* adds test for new query

* adds test data for new query

* adds ListModifiedSince stub to mockStorageBackend

* uncomment tests

* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.

* remove comments

* remove folder from query (dont need it, yet?)

* regen test queries

* updates test

* updates function comment

* use resourcepb.ResourceKey instead of ModifiedResourceKey

* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.

* skip TestListModifiedSince for kv backend

* use WatchEvent_Type for action type

* remove redundant fields from order by clause and regen test data for query

* remove redundant fields from order by clause and regen test data for query
2025-08-20 11:54:31 -06:00
Will Assis 34496e137c unified-storage: Search after write save rv to index (#109641)
* save rv to index after index is built
2025-08-19 09:41:35 -04:00
Will Assis 43cda1f1a4 unified-storage: disable watcher when search-after-write feature flag is enabled (#109634)
* disable indexer watcher when search-after-write feature flag is enabled
2025-08-14 10:18:09 -04:00
Ryan McKinley d3df5b8ddd Secrets: Manage secure values inside any resource (#107803) 2025-08-14 12:31:24 +00:00
Mustafa Sencer Özcan 7b4fcd486a Revert "fix: unified resource server list queries order column" (#109529) 2025-08-12 17:29:50 +02:00
Peter Štibraný 298b492f1a search queue index mutex (#109474)
Use custom mutex for index, and don't hold it during BulkIndex.
2025-08-11 16:20:47 +00:00
Ryan McKinley e0404f924c K8s/SecureValues: Wire InlineSecureValueSupport to apistore (#109449)
* inline wire

* extra fields

* add variable

* wire
2025-08-11 15:22:56 +03:00
owensmallwood c07c0f27d2 Unified Storage: Makes writing and reading index thread-safe in the queue processor (#109420)
makes writing and reading index thread-safe
2025-08-08 12:53:01 -06:00
owensmallwood e667dfed29 Unified Storage: Update index when indexqueueprocessor exists (#109351)
* Update readme

* When queue processor already exists, update the index on it in case it has been closed.

* Adds regression test.
2025-08-07 11:38:24 -06:00
Bruno Abrantes f3b8a891af chore: use native histogram for search shadow traffic match percentage (#109189)
Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-08-06 08:56:59 +02:00
Bruno Abrantes cb921dc47a feat: compare legacy and unified search results via histogram (#109022)
* feat: compare legacy and unified search results via histogram

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: handle cases where request type is not set

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: use struct instead of bool because it's more memory efficient

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

* fix: calculate recall percentage rather than union between legacy and unified

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>

---------

Signed-off-by: Bruno Abrantes <bruno@brunoabrantes.com>
2025-08-05 11:17:32 +02:00
Peter Štibraný 372f115db9 Finish building index even if original context is canceled. (#108969) 2025-07-31 13:45:55 +02:00
Peter Štibraný c28b2215e0 Improve indexing observability (#108901)
* Add details to tracing spans when creating index.

* Log reason for building index.

* Log reason for building index.

* Remove initialization of labels to avoid unnecessary metrics.

* Track succcessful, failed and skiped index builds. Track index build time for individual index, not all indexes.

* Revert removal of labels initialization.
2025-07-30 16:34:15 +02:00