* first pass of adding quotas service resource server
* passes prom reg as param
init quota service as part of server params
* init quota service as part of server params
* adds config and only creates quota service when overrides file path is defined
* when quota service enabled, check quota on create and log result
* update log message
* adds tests for quota service
* adds tests for config reloading when the file changes
* fix linter errors
* fix comment
* use startAndAwaitRunning
* Simplifies quotas service. Call manager.GetConfig() when getting quota instead of watching for changes.
* adds tracing to quotas service
* adds nsr attributes to traces when getting quotas and resource stats
* update comment
* update comment
remove check for nil overrides since it will (should) never happen
* fix linter error
* refactors naming to overrides service
checks quotas in separate function
* fix quotas naming
* fixes more quotas -> overrides naming
* use logger from ctx
* linter - remove trailing whitespace
* log FromContext() when checking quotas
* adds events to spans instead of create new spans
updates tenant -> namespace naming
few other minor fixes
* Unified Storage: allow rebuilding indexes for resource from a new grpc endpoint
* remove log line
* fix trace def
* lint
* fix after rebase
* addressing code review changes
* update with one channel per rebuild request
* other review suggestions
* update with review suggestions
* run mockery generate for MockResourceClient
* update tests
* update tests and lint
* fix test
* passes nsr to GetResourceStats instead of just namespace
* removes ns check
* fixes failing tests
* make update-workspace
* pass group and resource from rebuild request when getting resource stats
* LastImportTime for resource.
* Make StorageBackendImpl implement GetResourceLastImportTimes
* More missing implementations of GetResourceLastImportTimes
* Fix import.
* Skip TestGetResourceLastImportTime in TestBadgerKVStorageBackend.
* Implement GetResourceLastImportTimes by mockStorageBackend
* Bump test tolerance.
* Fix postgres query and timezone.
* Fix postgres query and timezone.
* Make linter happy.
* Don't use transaction in ListModifiedSince.
To guarantee that we don't include events with RV > LatestRV, we include the check in SQL query instead.
* Fix integration test by converting SQL comments into template comments.
* Skip query when ListModifiedSince cannot return anything.
* Only save listRV if it's different than sinceRV. This saves a disk access if not needed.
* Add test for ListModifiedSince with same RV.
* WIP added ListSinceModified to StorageBackend interface
* fix compile time check
* Fix method name
* Fix naming
* fix the rest of the ListSinceModified names
* Uses resource key without name field
* get latest rv from resource_version. Update test.
* adds moar tests
* adds method stub for ListModifiedSince to other StorageBackend implementations
* adds dummy impl to noop storage backend for ListModifiedSince
* skip tests for badger kv backend for now
* fixes tests and adds badgerkv impl for ListModifiedSince
* add badger kv impl
* adds test for new query
* adds test data for new query
* adds ListModifiedSince stub to mockStorageBackend
* uncomment tests
* refactors ListModifiedSince to return an iter.seq2 and handles deduplication. Updates tests. Updates query result sorting.
* remove comments
* remove folder from query (dont need it, yet?)
* regen test queries
* updates test
* updates function comment
* use resourcepb.ResourceKey instead of ModifiedResourceKey
* wrap seq in single transaction. Rollback transaction after 30s if iterator never used. Only track last seen event. Formatting.
* skip TestListModifiedSince for kv backend
* use WatchEvent_Type for action type
* remove redundant fields from order by clause and regen test data for query
* remove redundant fields from order by clause and regen test data for query
* make cgo optional for sqlite
* update go.mod; check error code differently
* reduce api surface even more
* move test errors into sqlite package
* add a comment
* Add max count config for indexing
* Build empty index when max count is exceeded
* Address linting
* Refactor buildIndexes
* Add test for max count threshold
* Update test doc comments
* Refactor TestBuildIndexes_MaxCountThreshold to not use mock framework
* Rename mocks used in TestBuildIndexes_MaxCountThreshold
* Refactor mockResourceIndex
* Test setting of indexing threshold configs
* Tweak comments, log
* Fix logging in buildEmptyIndex
* Export and reuse TestDocumentBuilderSupplier
* Reuse MockResourceIndex
* Use logging from app sdk in the resource mod
* make update-workspace
* Use app-sdk logging in SQL backend
* Use grafana-app logging in tests
* make update-workspace
* make update-workspace
* make update-workspace
* Use default logging
* Remove dependency on grafana/grafana
* Fix imports
* Unified Storage: Don't read before create
* test: use the existing test infra
* fix: support pq
We use pgx, but it seems to be wrapped in a pq driver shim, causing the errors to be remapped to pq's type. Weird
situation.
* feat: support CDK backend
* revert: there is a postgres_tests block
* fix(CDK): only check existence on ADDED updates
* fix(CDK): use ReadResource to deal with deleted files
* Unified Storage: Return an already exists error
When inserting a resource that already exists (i.e. race condition), we can safely catch UNIQUE constraint violations
and transform them into a `k8s.io/apimachinery/pkg/api/errors` error that stands the test of `errors.IsAlreadyExists`.
* feat: clarify existing conflict error
* chore: make update-workspace
* feat: make new package for backend error
* fix: assign dependency owner
* feat: use dialect for checking error type
* chore: go generate
* revert: to 5af369166d
* Add support for ASC ordering and introduce ResourceVersionMatch_Unset as default
Add SortAscending to continue token and add integration test for pagination.
* Change protobuf order
* Make backwards compatible
* Update pkg/storage/unified/sql/backend.go
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
---------
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
Co-authored-by: Jean-Philippe Quéméner <JohnnyQQQQ@users.noreply.github.com>
Add NotOlderThan support to getHistory
Add support for Exact
Add tests
Refactor tests
Add error test
Co-authored-by: Marco de Abreu <18629099+marcoabreu@users.noreply.github.com>
* move prometheus.register for unified storage metrics into metrics.go and do most of the plumbing to get it to work
* convert StorageApiMetrics to pointer and check for nil before using it
* rename type and variables to something more sensible
---------
Co-authored-by: Jean-Philippe Quéméner <jeanphilippe.quemener@grafana.com>
Will Assis
Jean-Philippe Quéméner
owensmallwood
João Calisto
Rafael Bortolon Paulovic
Peter Štibraný
Georges Chaudy
Ryan McKinley
Mustafa Sencer Özcan
Serge Zaitsev
Arati R.
Stephanie Hingtgen
Dana Axinte
Leonor Oliveira
Mariell Hoversholm
Marco de Abreu