Commit Graph
52 Commits
Author SHA1 Message Date
owensmallwood 5e3e7cca0e Search PoC: Adds traces to indexer (#95372)
* adds traces to indexer

* fix linting errors
2024-10-25 00:05:33 +03:00
owensmallwood ecebaf1bdf Unified Storage: Add traces to poller (#95369)
add traces to poller
2024-10-24 22:49:38 +03:00
Diego Augusto Molina 257aca5340 Unistore Chore: Fix span name (#94965)
fix span name
2024-10-24 12:12:20 -03:00
830600dab0 AuthN: Optionally use tokens for unified storage client authentication (#91665)
* extracted in-proc mode to #93124

* allow insecure conns in dev mode + refactoring

* removed ModeCloud, relying on ModeGrpc and stackID instead to discover if we're running in Cloud

* remove the NamespaceAuthorizer would fail in legacy mode. It will be added back in the future.

* use FlagAppPlatformGrpcClientAuth to enable new behavior, instead of legacy

* extracted authz package changes in #95120

* extracted server side changes in #95086

---------

Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
Co-authored-by: gamab <gabriel.mabille@grafana.com>
Co-authored-by: Dan Cech <dcech@grafana.com>
2024-10-24 09:12:37 +02:00
owensmallwood 2ffb88b0ee Unified Storage: Add index for poller query (#95293)
add index for poller query
2024-10-23 15:40:56 -06:00
Gabriel MABILLEandClaudiu Dragalina-Paraipan b68b69c2b4 AuthN: Use tokens for unified storage server authentication (#95086)
* Extract server code

---------

Co-authored-by: Claudiu Dragalina-Paraipan <drclau@users.noreply.github.com>
2024-10-23 15:04:15 +02:00
owensmallwood 9763199b53 Search Poc: Add metrics (#95111) 2024-10-22 12:25:08 -06:00
Diego Augusto Molina 3e1f5559a6 Unistore Chore: Add database-level observability (#92266)
* add testing harness

* fix mockery and linters

* WIP

* wip

* fix transactions

* fix transaction tracing; add tracing by default

* rename package

* move WithTx to simplify logic of DB implementations

* fix potential issue with context deadline

* add db instrumentation to dbutil

* add otel tests

* improve naming

* minor fix in semantics and add comprehensive OTel testing

* fix naming

* instrument resourceVersionAtomicInc

* provide a default testing tracer

* fix docs

* fix typo in docs

* add semconv for k8s
2024-10-18 00:32:08 -03:00
Diego Augusto Molina cf08f6762d Unistore Chore: Add OTEL testing harness (#94835)
* add testing harness

* fix mockery and linters

* fix data race in tests

* fix data race in tests

* reduce cardinality of data
2024-10-17 08:41:06 -03:00
Ryan McKinleyandJean-Philippe Quémémer 3457f219be Storage: Add blob storage interfaces (#90932)
Co-authored-by: Jean-Philippe Quémémer <jeanphilippe.quemener@grafana.com>
2024-10-17 12:18:29 +02:00
Diego Augusto Molina 44282134da Unistore Chore: Make it easier to implement DB interface (#94680)
make it easier to implement DB interface
2024-10-16 13:40:35 -03:00
Georges ChaudyandDiego Augusto Molina d999b415df UniStore: Use epoch with microsecond resolution as RV (#92638)
* Use epoch with microsecond resolution as RV

* fix backend tests

* Add solution for when the clock goes back

* Add solution for when the clock goes back

* generate mocks

* go lint

* remove comment

* Use Greatest instead of max in msyql and postgres

* update tests

* Update pkg/storage/unified/sql/sqltemplate/dialect_sqlite.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* cast to bigint

* add additional round trip

* increment the RV using 2 sql round trips instead of 3

* cleanup comments

* cast unix timestamp to integer

* fix postgres query

* remove old increment test data

* remove greatest

* cast unix_timestamp to signed

* Use statement_timestamp instead of clock_timestamp

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-10-11 12:11:33 +03:00
Prem Saraswat 5c03c14b25 resource-api: Loosen name validation to match K8s requirements (#93404)
* resource-api: Loosen name validation to match K8s requirements

This patch modifies some of the requirements for name validation of
objects in Resource API to match Kubernetes.

The limit we have on characters in name is 64, but some resources allow
upto 253 characters. Similarly we also include `:` in the regex, as many
objects in default K8s setup use it in the name (the group
`system:masters` for example)

Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>

* Update the name column length in migrator and update e2e test to verify

---------

Signed-off-by: Prem Kumar <prem.saraswat@grafana.com>
2024-10-10 11:03:18 -04:00
Diego Augusto Molina 4a800eda9f Unistore Chore: add inocuous preparative changes for otel-based db observability (#94473)
add inocuous preparative changes for otel tracing
2024-10-09 20:32:09 +00:00
Scott Lepper 52b70ca976 Grafana indexing poc - load and watch index (#93843)
* list resources and load into index
* watch for changes and update index
2024-10-08 09:43:23 -04:00
Arati R. ea5cf7c51f Unified Storage /Folders: Allow Unified Storage subfolders creation (#94327)
* Add parents field to folder DTO
* Allow subfolder creation when folder flag is enabled
* Update UnstructuredToLegacyFolder
* Include parents field when creating folder
2024-10-07 16:48:56 +02:00
Georges Chaudy 03f55e5288 Unistore : Ensure Watch works in HA mode (#94120)
* Revert "Revert "Unistore : Ensure Watch works in HA mode." (#94097)"

This reverts commit 7c3fc2f261.

* make previous_resource_version nullable

* handle nil case
2024-10-07 10:01:53 +02:00
Diego Augusto Molina 7d32d5eff4 Unistore: Reuse MySQL and Postgres Grafana core config instead of the object (#94223)
* Reuse MySQL and Postgres Grafana config instead of the object

- Only reuse the Grafana DB object for SQLite. Support for SQLite will be added in a different PR
- Fail when reusing the Grafana DB object if it is using DB instrumentation
- In the case that we have to reuse a Grafana DB with its instrumentation, fail with an error that describes a workaround
- Add regression tests to reproduce incident 2144

* remove temp file

* fix linter

* fix linter x2

* fix linter x3
2024-10-04 12:07:20 +00:00
Georges Chaudy 7c3fc2f261 Revert "Unistore : Ensure Watch works in HA mode." (#94097)
Revert "Unistore : Ensure Watch works in HA mode. (#93428)"

This reverts commit 0a26c9e9ae.
2024-10-01 18:45:47 +00:00
6a3eb276ef Grafana Indexing PoC: Adds feature flag and gRPC endpoint (#93356)
* adds Filter gRPC and make protobuf

* adds route for querying the filter gRPC

* wires up Filter gRPC call

* [WIP] index from start

* renames gRPC endpoint to "Search"

* adds /apis/search route into k8s routes. Hacky for now.

* updates readme - wrong casing

* adds feature toggle for unified storage search

* hides US search behind feature flag. Clean up print statements.

* removes indexer - will be added in another PR

* Search: Add API Builder

* adds required method

* implementing UpdateAPIGroupInfo (WIP)

* adds groupversion

* commenting out for now

* remove unneeded code from experimenting and update register.go to match interface required

* namespaces search route

---------

Co-authored-by: leonorfmartins <leonorfmartins@gmail.com>
Co-authored-by: Todd Treece <todd.treece@grafana.com>
2024-09-30 13:46:14 -06:00
Georges ChaudyandDiego Augusto Molina 0a26c9e9ae Unistore : Ensure Watch works in HA mode. (#93428)
* Replace Watch with WatchNext

* remove watchset

* fix previous page and closing the channel

* Remove the broadcaster cache to prevent dupplicated events

* add watch bookmark

* add watch bookmark

* cleanup comments

* disable the tests for bookmarks for now

* Ensure we send previosu events

* lint

* re-introduce the cache

* load from cache

* disabling legacy test

* disabling legacy test

* Update pkg/storage/unified/resource/server.go

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>

* Could not read previous events

* add proper migration

* Add previous_resource_version to both history and resource

* First event should have an RV of 2 and not 1

* Test both storage backends

* fix the inital RV for the sql backend

* ensure graceful stop of the stream decoder

* gocyclo

---------

Co-authored-by: Diego Augusto Molina <diegoaugustomolina@gmail.com>
2024-09-30 13:14:07 +02:00
Georges Chaudy 216b63549d unistore: fix watch events forever looping (#93517) 2024-09-20 15:51:09 +02:00
Ryan McKinley 542105b680 ResourceClient: Exercise resource client in k8s apis tests (#93473) 2024-09-19 17:16:48 +03:00
Ryan McKinley 45eb72e95a K8s: Use wire to initalize the resource client (#93221) 2024-09-12 17:22:27 +03:00
Ryan McKinley 7efadb0a00 Testing: Integration tests for unified-grpc start a local gRPC server (#93201) 2024-09-11 11:50:14 +03:00
Ryan McKinley 5441e4c752 Storage: create a client with access to all interfaces, not just ResourceStore (#92967) 2024-09-05 12:52:30 +03:00
Ryan McKinley c59dddf7af MySQL: Add parseTime=true to SQL connections (#92469) 2024-08-27 14:16:04 +03:00
Ryan McKinley 5a30e12a10 SQLTemplate: Make Ident only work for identifiers (not any string) (#92387) 2024-08-27 13:22:40 +03:00
Ryan McKinley 2136fd9a92 Storage: Remove unified storage feature flag (#92192)
remove unified storage flag
2024-08-21 19:28:30 +03:00
Dave HendersonandRyan McKinley df3d8915ba Chore: Bump Go to 1.23.0 (#92105)
* chore: Bump Go to 1.23.0

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* update swagger files

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore: update .bingo/README.md formatting to satisfy prettier

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* chore(lint): Fix new lint errors found by golangci-lint 1.60.1 and Go 1.23

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

* keep golden file

* update openapi

* add name to expected output

* chore(lint): rearrange imports to a sensible order

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>

---------

Signed-off-by: Dave Henderson <dave.henderson@grafana.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-08-21 11:40:42 -04:00
Diego Augusto Molina 9a65d0a977 Storage: remove sqlstore.WrapDatabaseDriverWithHooks (#92111)
remove sqlstore.WrapDatabaseDriverWithHooks from Unified Storage after related incident
2024-08-21 10:31:34 -03:00
Diego Augusto Molina e788df921c Storage: always use transactions and make them readonly when possible (#92110)
* always use transactions and make them readonly when possible

* fix linters

* fix reference
2024-08-20 09:29:06 -03:00
owensmallwood 1cb3470e3e Unified Storage: Wrapping the mysql driver with hooks causes issues (#92099)
dont wrap the mysql driver with hooks. Its causing the error 'begin tx: sql: driver does not support non-default isolation level'.
2024-08-19 15:09:25 -06:00
Ryan McKinley f432a1713b SQLTemplate: Swap the IFace with the concrete struct (#92008) 2024-08-16 15:12:37 +03:00
Ryan McKinley d9cabe5e14 SQL: Add sql template test helper (#91953) 2024-08-16 14:36:56 +03:00
owensmallwood 28f2a4078d Unified Storage: Records grafana database metrics (#91932)
* records grafana database metrics for unified storage

* update type to tracing.Tracer

* use nil for tracer in tests
2024-08-15 11:13:36 -06:00
Ryan McKinley 21d4a4f49e Auth: use IdentityType from authlib (#91763) 2024-08-12 09:26:53 +03:00
Ryan McKinley 9e116d13a5 K8s/Dashboards: Fix dashboard list and add tests (#91523) 2024-08-07 13:43:13 +03:00
Georges Chaudy 36c2e0cfae unifiedStorage: sort by namespace/name ASC (#91314)
* unified: sort by key asc

* sort by key

* keep tests commented
2024-08-01 08:27:01 +00:00
Ryan McKinley f804b0baa3 Storage: Return an iterator from backend List (#91185) 2024-07-31 12:05:59 +03:00
Ryan McKinley 160fe2a3a4 Storage: Remove entity store (#91233) 2024-07-31 09:25:39 +03:00
Diego Augusto Molina 30d4a4249c Storage: Disable Resource Server integration tests to unblock pipelines (#91216)
disable integration tests to unblock pipelines until we find the issue
2024-07-30 21:39:33 -03:00
owensmallwood 2362e79e4e Unified Storage: Configure client to use tls=preferred (#91162)
* optionally set tls=preferred in the mysql connection string from env var
2024-07-30 11:24:28 -06:00
Georges Chaudy 901f7f1529 unifiedStorage: fix flaky integration test (#91199)
unified: fix flaky integration test
2024-07-30 16:55:51 +03:00
Ryan McKinley da65825c5f Storage: Consolidate error handling (#91167) 2024-07-30 10:16:16 +00:00
Georges Chaudy 7ad37ff4e4 UnifiedStorage: ensure list from history works across all 3 database backends (#91134)
* Ensure List is sorted by resource version DESC

* Ensure List is sorted by resource version DESC

* fix list in the past for postgres
2024-07-29 13:49:54 -06:00
Georges Chaudy e4ed000d05 ResourceStore: Fix sql List (#91043)
* fix resource list

* ensure we get the resource version without FOR SHARE
2024-07-26 13:56:24 +02:00
Georges ChaudyandRyan McKinley 4baca6947d ResourceServer: make the resource store the default unified storage backend (#90899)
* make the resource store the default unified storage backend

* add integration tests

* fix test non passing

* Update pkg/storage/unified/sql/test/integration_test.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* lint

* fix tests

* fix no rows

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2024-07-25 18:17:39 +02:00
owensmallwood 8fd2245ce6 Grafana: Skip US tests breaking enterprise for now (#90869)
* skip US tests breaking enterprise for now

* skips rest of tests
2024-07-23 16:08:47 -06:00
Diego Augusto Molina 399d77a0fd Resource server improvements and fixes (#90715)
* cleanup dependencies and improve list method
* Improve Resource Server API, remove unnecessary dependencies
* Reduce the API footprint of ResourceDBInterface and its implementation
* Improve LifecycleHooks to use context
* Improve testing
* reduce API size and improve code
* sqltemplate: add DialectForDriver func and improve naming
* improve lifecycle API
* many small fixes after adding more tests
2024-07-22 20:08:30 +03:00