Commit Graph

379 Commits

Author SHA1 Message Date
Artur Wierzbicki 736be74128 Chore: Add Usage stats providers registry (#48357)
* add usage stats providers

* migrate thumbnails service to `registry.ProvidesUsageStats`

* snake -> camel

* lint fix

* migrate accesscontrol to `registry.ProvidesUsageStats`

* add accesscontrol to usage stats providers registry

* fix test

* added a note about errors
2022-04-28 13:06:49 +04:00
Guilherme Caulada a367ad730c Secrets: Implement basic unified secret store service (#45804)
* wip: Implement kvstore for secrets

* wip: Refactor kvstore for secrets

* wip: Add format key function to secrets kvstore sql

* wip: Add migration for secrets kvstore

* Remove unused Key field from secrets kvstore

* Remove secret values from debug logs

* Integrate unified secrets with datasources

* Fix minor issues and tests for kvstore

* Create test service helper for secret store

* Remove encryption tests from datasources

* Move secret operations after datasources

* Fix datasource proxy tests

* Fix legacy data tests

* Add Name to all delete data source commands

* Implement decryption cache on sql secret store

* Fix minor issue with cache and tests

* Use secret type on secret store datasource operations

* Add comments to make create and update clear

* Rename itemFound variable to isFound

* Improve secret deletion and cache management

* Add base64 encoding to sql secret store

* Move secret retrieval to decrypted values function

* Refactor decrypt secure json data functions

* Fix expr tests

* Fix datasource tests

* Fix plugin proxy tests

* Fix query tests

* Fix metrics api tests

* Remove unused fake secrets service from query tests

* Add rename function to secret store

* Add check for error renaming secret

* Remove bus from tests to fix merge conflicts

* Add background secrets migration to datasources

* Get datasource secure json fields from secrets

* Move migration to secret store

* Revert "Move migration to secret store"

This reverts commit 7c3f872072.

* Add secret service to datasource service on tests

* Fix datasource tests

* Remove merge conflict on wire

* Add ctx to data source http transport on prometheus stats collector

* Add ctx to data source http transport on stats collector test
2022-04-25 13:57:45 -03:00
Artur Wierzbicki 9f0b6a5754 Storage: refactor filtering, improve performance (#47403)
* Storage: refactor filtering, improve performance

* added a comment to `newpathfilter`

* after merge fixes
2022-04-21 23:27:43 +04:00
Emil Tullstedt d76e5d7c6a Usage stats: Detect Prometheus flavors (#47942)
* Naïve Prometheus flavor detector

* Add concurrency and memoization

* Remove concurrency

* Fix tests

* close response body

* Add tests
2022-04-20 13:51:33 +00:00
Serge Zaitsev 41012af997 Tracing: Use common traceID context value for opentracing and opentelemetry (#46411)
* use common traceID context value for opentracing and opentelemetry

* support sampled trace IDs as well

* inject traceID into NormalResponse on errors

* Finally the test passed

* fix the test

* fix linter

* change the function parameter

Co-authored-by: Ying WANG <ying.wang@grafana.com>
2022-04-14 17:54:49 +02:00
ying-jeanne f62c261900 fix terminal format (#47718) 2022-04-13 17:37:56 +02:00
ying-jeanne 06d2d44031 put the default logger format and feature toggle to fallback to old logger (#47584) 2022-04-13 11:02:19 +02:00
Artur Wierzbicki 712b239d5a Storage: remove pointer on write path (#47580)
* remove pointer to contents in UpsertFileCommand

* add command to UpsertFileCommand

* fix client after API refactor
2022-04-12 16:58:09 +02:00
ying-jeanne 0bf889e058 Fix unmaarshal of double pointer (#47586)
* Fix unmaarshal of double pointer

* update sdk version
2022-04-12 09:30:34 +02:00
ying-jeanne 03ba91e8a4 add feature toggle with new format (#47336)
* add feature toggle with new format

* fix some comments ❤️

* Update pkg/infra/log/term/terminal_logger.go

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>

Co-authored-by: Emil Tullstedt <emil.tullstedt@grafana.com>
2022-04-11 18:50:29 +02:00
Emil Tullstedt 3df625e9f4 UsageStats: Move stats collection to separate service (#47381)
* Remove specific stats from usage stats service

* Create statscollector service

* refactor

* Update and move tests

Mostly equivalent tests to before, but they've been divided over the two
services and removed the behavior driven legacy from GoConvey to
reduce the complexity of the tests.

* Collect featuremgmr metrics (copied over from #47407)

I removed the metrics registration from the feature manager in the merge
and re-add them in this commit. Separated to make things easier to
review.
2022-04-08 13:41:26 +02:00
Artur Wierzbicki f1a1070d41 UsageStats: track enabled features (#47407)
* #47127: include enabled features in the usage stats reports

* #47127: convert feature names to snake cased metric names

* #47127: remove dead code

* #47127: lint fix

* #47127: convert GetUsageStats to return `map[string]interface{}`

* #47127: fix testssssssss

* #47127: fix testssssssss
2022-04-08 12:42:33 +02:00
ying-jeanne c525db9cd8 fallback to the v8 logging behavior (#47353) 2022-04-05 20:15:41 +02:00
Marcus Efraimsson 659cf17e10 Logging: Fix syslog messages should be sent with correct severity (#47209) 2022-04-04 12:51:14 +02:00
Marcus Efraimsson b4346a5613 Logging: Fixes filtering logs based on gokitlog levels (#47195) 2022-04-04 12:36:17 +02:00
Sergey Kostrukov 656ade9884 Shared Azure middleware between Azure Monitor and Prometheus datasources (#46002)
* Scopes in Azure middleware

* Enable Azure middleware without feature flag

* Use common Azure middleware in Azure Monitor

* Apply feature flag to JsonData configuration of Azure auth

* Enforce feature flag in Prometheus datasource

* Prometheus provider tests

* Datasource service tests

* Fix http client provider tests

* Pass sdkhttpclient.Options by reference

* Add middleware to httpclient.Options

* Remove dependency on Grafana settings

* Unit-tests updated

* Fix ds_proxy_test

* Fix service_test
2022-04-01 13:26:49 +02:00
Marcus Efraimsson 16db1ad46d API: Fixes request logging middleware to include userId, orgId, uname (#47183)
Fixes #47006
2022-04-01 13:24:39 +02:00
Emil Tullstedt 7bc821baf2 Logger: Use specified format for file logger (#46970)
Fixes #45941
2022-03-26 11:40:08 +01:00
Eng Zer Jun b56848f006 test: use T.TempDir to create temporary test directory (#44947)
The directory created by `T.TempDir` is automatically removed when the
test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-03-22 15:43:29 +01:00
Ryan McKinley 1cfb9a4a19 Storage: add basic storage service (#46604) 2022-03-17 10:19:23 -07:00
Jguer d5883c1b27 Service Accounts: Implement basic usage stats (#46619)
* Stats: do not count SAs as users

* Stats: implement basic service account metrics

* Stats: do not count service account tokens as api keys

* Stats: fix metric names

* Stats: add SA stats test

* rename user to sa
2022-03-16 16:54:34 +01:00
Artur Wierzbicki b8fba41d74 Storage: unify List queries (#46572)
* silence errors

* s3 fix - don't retrieve files with path equal to the root

* Storage: unify list queries

* Storage: add `IsFolder` method to file obj

* Storage: API consistency - always refer `File` as a pointer rather than a value
2022-03-15 21:21:22 +04:00
Artur Wierzbicki 5881127319 Storage: fix filestorage.Wrapper PathFiltering and non-recursive folder listing (#46469)
* fix pathFilters in wrapper - apply rootPath changes before filtering

* fix non-recursive folder listing
2022-03-11 23:16:46 +04:00
Artur Wierzbicki ed924b3d0c Storage: list filtering and root folder support (#46453)
* git the things: FS api internal changes

* remove filestorage/service.go

* remove filestore flag

* remove dummy fs

* readd fileblob import
2022-03-11 22:08:19 +04:00
gotjosh b1e6f7126a Logging: Helper to create NopLogger (#46376) 2022-03-09 17:03:58 +00:00
Artur Wierzbicki a8b90d9a25 FileStore: add basic file storage API (#46051)
* #45498: fs API alpha

* #45498: remove grafanaDS changes for filestorage.go

* #45498: fix lint

* #45498: fix lint

* #45498: remove db file storage migration

* #45498: linting

* #45498: linting

* #45498: linting

* #45498: fix imports

* #45498: add comment

* remove StorageName abstractions

* FileStore: add dummy implementation (#46071)

* #45498: bring back grafanaDs changes, add dummy filestorage

* #45498: rename grafanaDs to public

* #45498: modify join

* #45498: review fix

* #45498:  unnecessary leading newline (whitespace) IMPORTANT FIX

* #45498: fix belongsToStorage

* #45498: fix removeStoragePrefix so that it works with abs paths

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
2022-03-03 10:53:26 +04:00
Selene 2e6f14d17f Bus: Tests cleanup (#46025)
* Delete unused bus from usagestats

* Few updates to remove bus from searchusers test

* Fix import
2022-03-02 11:05:31 +01:00
idafurjes d8a56d08ba Remove bus from usage stats (#45275) 2022-02-11 14:04:15 +01:00
Marcus Efraimsson ca311bf94f Logging: Fix server hangs when using log.WithSuffix (#45213)
Fixes a problem where server hangs if setting 
GF_DATABASE_LOG_QUERIES=true.

Ref #44925
2022-02-10 12:51:51 +01:00
Marcus Efraimsson 2082eeb72f Logging: Fix so that filters can contain commented lines (#45159)
Fixes log filters that contains comments should not be enabled
2022-02-10 09:52:37 +01:00
Will Browne fc42dfe396 Auth: Guarantee consistency of signed SigV4 headers (#45054)
* use latest sigv4 changes

* update configuration docs

* lint

* reformat lint ignore

* specific version for docs
2022-02-08 14:48:17 +01:00
Marcus Efraimsson 05ea825c76 Chore: Logging improvements (#44925)
Fixing a couple bugs, adds some tests and hopefully decrease 
lock contention when logging.
Switching from using sync.RWMutex to go-kit SwapLogger.
Fixes bug when creating a new logger from an existing one that 
screwed up the keyvals and/or lost the logger name. 

Ref #44681
2022-02-07 16:00:08 +01:00
bolshoytoster 7f2d3055d8 Logging: Fix don't return an error trying to initialize the file logger (#42952)
Fixes #42897

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-02-03 18:17:25 +01:00
Marcus Efraimsson bc7e55d99b Chore: Fix log filters (#44681) 2022-02-03 23:20:02 +08:00
Marcus Efraimsson a23900c4ba CodeQL: Ignore go/log-injection for terminal logger (#44481) 2022-01-27 19:01:59 +01:00
Ryan McKinley 3c334cd8ba Logger: avoid panic when time is a string (not fmt.Stringer) (#44497) 2022-01-26 13:17:42 -08:00
Ryan McKinley 5d66194ec5 FeatureFlags: define features outside settings.Cfg (take 3) (#44443) 2022-01-26 09:44:20 -08:00
Marcus Efraimsson 50fabe8a87 Chore: Implements log15 terminal logger (#44447)
Nicer development/terminal output based on log15 terminal logger.
2022-01-26 12:19:43 +01:00
Sergey Kostrukov cb09162cde Datasources: Simplify the AzureCredentials structure in datasource config (#39209)
Related #35857

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-01-25 14:23:32 +01:00
Agnès Toulet 65bdb3a899 FeatureFlags: Revert managing feature flags outside of settings.Cfg (#44382)
* Revert "FeatureToggles: register all enterprise feature toggles (#44336)"

This reverts commit f53b3fb007.

* Revert "FeatureFlags: manage feature flags outside of settings.Cfg (#43692)"

This reverts commit f94c0decbd.
2022-01-24 16:08:05 +01:00
Ryan McKinley f94c0decbd FeatureFlags: manage feature flags outside of settings.Cfg (#43692) 2022-01-20 13:42:05 -08:00
idafurjes 30aa24a183 Chore: Implement OpenTelemtry in Grafana (#42674)
* Separate Tracer interface to TracerService and Tracer

* Fix lint

* Fix:Make it possible to start spans for both opentracing and opentelemetry in ds proxy

* Add span methods, use span interface for rest of tracing

* Fix logs in tracing

* Fix tests that are related to tracing

* Fix resourcepermissions test

* Fix some tests

* Fix more tests

* Add TracingService to wire cli runner

* Remove GlobalTracer from bus

* Renaming test function

* Remove GlobalTracer from TSDB

* Replace GlobalTracer in api

* Adjust tests to the InitializeForTests func

* Remove GlobalTracer from services

* Remove GlobalTracer

* Remove bus.NewTest

* Remove Tracer interface

* Add InitializeForBus

* Simplify tests

* Clean up tests

* Rename TracerService to Tracer

* Update pkg/middleware/request_tracing.go

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>

* Initialize tracer before passing it to SQLStore initialization in commands

* Remove tests for opentracing

* Set span attributes correctly, remove unnecessary trace initiliazation form test

* Add tracer instance to newSQLStore

* Fix changes due to rebase

* Add modified tracing middleware test

* Fix opentracing implementation tags

Co-authored-by: Marcus Efraimsson <marcus.efraimsson@gmail.com>
2022-01-20 11:10:12 +01:00
J Guerreiro 1e9818a69f Add usage stats preview endpoint (#43899)
* Stats: add preview route for usage statistics

* Stats: respect reporting settings

* Stats: add tests to api endpoint

* Stats: unregister route

* Stats: always possible to preview stats

* Update pkg/infra/usagestats/service/api.go

Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>

Co-authored-by: Gabriel MABILLE <gamab@users.noreply.github.com>
2022-01-14 10:16:07 +01:00
Yuriy Tseretyan 8114f6b065 Use stack trace context in XORM trace logger (#43780)
* add caller and stack Valuer functions
* Add WithPrefix and WithSuffix similar to what go-kit offers
* replace New with just `with`. Remove filter wrapper because the first argument of the context argument is not logger but additional context.
* update Xorm logger to use custom depth to display the datastore code instead of xorm
2022-01-13 13:30:28 -05:00
ying-jeanne a8eef45a44 Logger migration from log15 to gokit/log (#41636)
* migrate log15 to gokit/log

* fix console log

* update some unittest

* fix all unittest

* fix the build

* Update pkg/infra/log/log.go

Co-authored-by: Yuriy Tseretyan <tceretian@gmail.com>

* general type vector

* correct the level key

Co-authored-by: Yuriy Tseretyan <tceretian@gmail.com>
2022-01-06 22:28:05 +08:00
idafurjes 8e6d6af744 Rename DispatchCtx to Dispatch (#43563) 2021-12-28 17:36:22 +01:00
idafurjes 7936c4c522 Rename AddHandlerCtx to AddHandler (#43557) 2021-12-28 16:08:07 +01:00
idafurjes 56c3875bb9 Chore: Remove context.TODO (#43458)
* Remove context.TODO() from services

* Fix live test
2021-12-28 10:26:18 +01:00
J Guerreiro a1b8b5d123 Usage stats: Count API keys (#42883)
* Stats: add api key count to usage stats

* Chore: correctly capitalize struct member
2021-12-22 17:37:45 +01:00
idafurjes b8852ef6a3 Chore: Remove context.TODO() (#43409)
* Remove context.TODO() from services

* Fix live test

* Remove context.TODO
2021-12-22 11:02:42 +01:00