Commit Graph

76 Commits

Author SHA1 Message Date
Sven Grossmann a01f8c5b42 Loki: Add backend functionality to parse structured metadata from Loki (#77361)
* prepare backend for structured metadata

* add `lokiStructuredMetadata` feature toggle

* use `lokiStructuredMetadata` feature flag

* add field type check to `labelTypesField`

* remove fixme

* fix feature toggle

* add field in dataplane mode

* use `data.Labels` where possible

* adjust framing tests

* improve verbiage

* improve naming

* update tests to "attributes"
2023-11-16 17:06:14 +01:00
Ryan McKinley f69fd3726b FeatureToggles: Add context and and an explicit global check (#78081) 2023-11-14 12:50:27 -08:00
Ivana Huckova 8cd8eb7882 Revert "Loki: Add error source to DataQuery (#77876)" (#78006)
This reverts commit 934456dc1c.
2023-11-10 17:03:59 +01:00
Ivana Huckova 934456dc1c Loki: Add error source to DataQuery (#77876)
* Loki: Add error source to DataQuery

* Create middlewares to fix healtcheck test

* Add comment for errors from ReadPrometheusStyleResult
2023-11-10 10:26:56 +01:00
Will Browne 9d92818cae Plugins: Update PDC pattern from latest plugin SDK changes (#76576)
* update with sdk

* do sql

* fix core plugins

* fix proxy settings

* bump SDK version

* tidy

* enable pdc for test

* add codeowners

* bump dep

* go mod tidy

* bump SDK
2023-10-16 16:40:04 +02:00
Dan Cech 91cf4f0c1c Revert "Plugins: Update PDC pattern from latest plugin SDK changes" (#76565)
Revert "Plugins: Update PDC pattern from latest plugin SDK changes (#76036)"

This reverts commit 151f6d6216.
2023-10-13 12:35:16 -04:00
Will Browne 151f6d6216 Plugins: Update PDC pattern from latest plugin SDK changes (#76036)
* update with sdk

* do sql

* fix core plugins

* fix proxy settings

* bump SDK version

* tidy

* enable pdc for test

* add codeowners
2023-10-13 14:42:15 +02:00
Marcus Efraimsson e4c1a7a141 Tracing: Standardize on otel tracing (#75528) 2023-10-03 14:54:20 +02:00
Will Browne 7fca1bde54 Plugins: Set grafana config, plugin version and user agent on plugin requests (#75171)
* first pass

* fixup

* remove test line

* fix tests

* use new fields

* fix imports + formatting

* fix tests

* rollback changes

* undo whitespace

* apply pr feedback
2023-09-21 11:33:31 +02:00
Travis Patterson 98aa7db64a Loki: Run queries in parallel behind lokiRunQueriesInParallel feature toggle (#74064)
* issue loki queries in parallel so total query time is only the slowest query rather than the sum of all query times.

* Fix lint

* Add running of queries in parallel behind feature toggle to test the functonality before release

* Add span end

* Move shared logic to separate function

* Add logging and tracing around running of all queries

---------

Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>
2023-09-19 12:34:01 +03:00
Sven Grossmann 727c83d562 Loki: Improve error logging in case statusCode is not 200 (#75075)
* improve loki logging

* fix error log

* fix logger arg
2023-09-19 11:11:27 +02:00
Sven Grossmann d14851c877 Loki: Adjust to logging convention (#74531)
* adjust Loki to logging convention

* Fix call resource logging

* Update dataquery

* Update

* Remove redundant logging

* Fix TODO

* Rename action to stage and use variables

* `resp` might be `nil`

* `resp` might be `nil` here as well

* change to `statusCode`

* use correct logger

* also here

* add query information to logging

---------

Co-authored-by: Ivana Huckova <ivana.huckova@gmail.com>
2023-09-07 17:09:24 +02:00
Sven Grossmann a403027608 Loki: Add backend healthcheck (#74330)
* add loki healthcheck

* remove `testDatasource` call

* remove unused error check

* change success message

* improve error messages
2023-09-05 08:59:13 +02:00
Serge Zaitsev 93cdc94a94 Chore: capitalise logs in other backend packages (#74344)
* capitalise logs in observability logs

* capitalise oss-bit-tent packages

* capitalise logs in aws-datasources

* capitalise logs for traces and profiling

* capitalise logs for partner datasources

* capitalise logs in plugins platform

* capitalise logs for observability metrics
2023-09-04 22:25:43 +02:00
Sven Grossmann 056c83c788 Loki: Improve backend instrumentation of CallResource calls (#73899)
* instrument call resource

* capitalize log messages
2023-09-01 15:26:25 +02:00
Sven Grossmann 124f445db0 Loki: Improve backend instrumentation of QueryData calls (#73807)
* improve loki `queryData` instrumentation

* fix lint

* use correct logger in "queryData"

* capitalize log messages

* distinguish between requests from alerting
2023-09-01 11:44:18 +02:00
Sven Grossmann 703f4be15d Loki: Fix wrong context being passed to HTTP client (#72991)
use new context containing tracing information
2023-08-08 10:24:07 +02:00
Gábor Farkas e045860fbf loki: generate dataplane-compliant logs dataframes (behind feature flag) (#69909)
* add feature flag

* prepare tests

* enable dataplane-format in loki-backend
2023-07-13 09:58:00 +02:00
Will Browne 286b9e08e9 Plugins: Bump Plugin SDK version and address instance management breaking changes (#68900)
* bump sdk and propagate ctx

* fix tests
2023-05-24 10:19:34 +02:00
Gábor Farkas 531caec602 Loki: Add feature flag to enable dataplane-compliant metric frames (#66017)
* loki: add feature flag to enable dataplane-compatible metric frames

* linter fix
2023-04-13 15:07:08 +02:00
Sven Grossmann 7261c6f7cd DataSourceWithBackend: Add queryGroupId to find correlated/related queries (#64587)
* add `correlationId` to queries

* trace correlation id in backend

* add correlation id to loki's span

* add correlation id to query chunks

* fix test

* fix DataSourceWithBackend test

* change to `queryGroupId`

* remove empty line

* fix test in `DataSourceWithBackend`
2023-03-17 15:48:13 +01:00
Sven Grossmann 473013e3f5 Loki: Add error handling to CallResource (#64439)
* surface loki error message for `CallResource`

* use `data.Message` instead or `errorMessage`

* change struct coming from Loki

* remove whitespace
2023-03-09 11:12:33 +01:00
Gábor Farkas 48a374f50b loki: implement schema for loki query (#62114)
* loki: implement schema for loki query

* removed forgotten code

* better types

* adjust cue-maturity-level

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* sync docs to schema change

* removed forgotten go code

* updated schema docs

Co-authored-by: J Stickler <julie.stickler@grafana.com>

* updated schema docs

Co-authored-by: J Stickler <julie.stickler@grafana.com>

* update schema docs

Co-authored-by: J Stickler <julie.stickler@grafana.com>

* update schema docs

Co-authored-by: J Stickler <julie.stickler@grafana.com>

* updated schema docs

Co-authored-by: J Stickler <julie.stickler@grafana.com>

* updated schema docs

* add documentation to the resolution field

* move direction-field to schema

* loki: cue: moved deprecated fields to the bottom

---------

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>
Co-authored-by: J Stickler <julie.stickler@grafana.com>
2023-02-02 16:18:30 +00:00
Gareth Dawson b2c8126e6e Loki: Show query size approximation (#62109)
* feat: make api request to /loki/api/v1/index/stats

* fix: add /index/stats to callResource valid urls

* feat: make call to getQueryStats when the query changes

* feat: render user tooltip displaying the estimated value for processed data

* fix: add new props to component tests

* test: add tests for query size estimation

* fix: disable error message on request failure

* refactor: add suggestions from code review

* refactor: only pass required query string
2023-01-27 17:32:53 +00:00
Ivana Huckova d0e95f8c95 Loki: Add X-Query-Tags header for logs sample and data sample (#62333) 2023-01-27 16:41:40 +01:00
Marcus Efraimsson c35c689a96 Plugins: Automatically forward plugin request HTTP headers in outgoing HTTP requests (#60417)
Automatically forward core plugin request HTTP headers in outgoing HTTP requests. 
Core datasource plugin authors don't have to specifically handle forwarding of HTTP 
headers, e.g. do not have to "hardcode" the header-names in the datasource plugin, 
if not having custom needs.

Fixes #57065
2022-12-21 13:25:58 +01:00
Yann Vigara f1ef63791a Fix #58598 X-ID-Token header missing on Loki Datasource (#58784)
* Fix #58598 X-ID-Token header missing on Loki Datasource

* Remove unecessary continue statements

* Add getAuthHeadersForCallResource unit tests

* Fix test and switch statement issues introduced during merge
2022-11-23 11:31:50 +01:00
Sven Grossmann 08e87a217c Loki: Add gzip compression to resource calls (#59059)
* Loki: Add compression to `callResource`

* add missing tests

* fix formatting
2022-11-22 14:02:30 +01:00
Si Mon bc280d0749 Datasource Loki: preserve header X-ID-Token (#57878) 2022-11-07 16:53:42 +01:00
Yuriy Tseretyan fa043efb2c Fix: correctly close trace span in Prometheus and Loki data sources (#57774)
* rename span for loki
2022-10-28 10:17:56 -04:00
Yuriy Tseretyan facf2b1ee8 Chore: Update prometheus, loki, graphite and influx plugins to support contextual logs. (#57708) 2022-10-27 12:05:06 -04:00
Alexander Weaver 145b970557 Propagate all headers on main QueryData (#56896) 2022-10-13 14:18:53 -05:00
Gábor Farkas 8fd9cb4854 loki: add feature-toggle check to live-mode (#49012) 2022-06-13 08:33:46 +02:00
Gábor Farkas 46d6573968 loki: add cookie-handling functionality (#49978) 2022-06-02 11:52:27 +02:00
Gábor Farkas da74dba7c8 Loki: backend: use streaming JSON parser, try2 (#48752)
* converter: remove __name__ customization because Loki does not do that

Loki does not handle __name__ in a special way.
for Prometheus, the caller can implement the formatting
by themselves

* converter: change labels-formatting

the labels.String() method does not handle strange values well

* loki: backend: use streaming-json parser

* more idiomatic code

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* simpler row-length check

* simpler code

* fixed converter/prom tests

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
2022-05-05 13:09:01 +02:00
Gábor Farkas 02aa1cd1c5 loki backend mode forward-oauth (#48401)
* loki: backend: add forward oauth credentials functionality

* removed obsolete comment
2022-05-05 12:42:50 +02:00
Gábor Farkas 3cade2f669 Revert "Loki: backend: use streaming JSON parser (#47656)" (#48747)
This reverts commit 46b40b6e82.
2022-05-05 12:02:53 +02:00
Gábor Farkas 46b40b6e82 Loki: backend: use streaming JSON parser (#47656)
* converter: remove __name__ customization because Loki does not do that

Loki does not handle __name__ in a special way.
for Prometheus, the caller can implement the formatting
by themselves

* converter: change labels-formatting

the labels.String() method does not handle strange values well

* loki: backend: use streaming-json parser

* more idiomatic code

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>

* simpler row-length check

* simpler code

Co-authored-by: Todd Treece <360020+toddtreece@users.noreply.github.com>
2022-05-05 08:47:40 +02:00
Gábor Farkas ae926c2144 loki: better resource-call urls (#48250) 2022-04-28 14:28:57 +02:00
Gábor Farkas c73b9d1818 Loki: do not use the old labels-api (#48185)
* loki: do not use old labels-api

* adjust allowed-url-list

* fixed tests
2022-04-26 11:51:26 +02:00
Gábor Farkas d8a754c4a0 loki: send metadata requests through backend (#48063) 2022-04-25 13:16:14 +02:00
Gábor Farkas 79c06fdddc Loki: add backend-forward mode to queries, update log-row-context (#47726)
* loki: add helper function to sort dataframe by time

* loki: add direction-attribute to queries

* loki: make log-row-context code backward-compatible

* better comment

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* fixed test

* simplified code

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
2022-04-20 13:52:15 +02:00
Ryan McKinley 796bc27f75 Loki: support live streaming (#42804) 2022-03-01 14:46:52 -08:00
Gábor Farkas 1cad35ea67 loki: backend mode: support all query types (#45619)
* loki: backend mode: support all query types

* loki: backend: adjust vector-parsing field-names

* loki: backend: no interval for streams-dataframes

* loki: backend: enable more query types

* better variable name

* removed unnecessary code

* improve frame-processing

* more unit tests

* improved code

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* remove unused code

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* simplify code

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

* lint fix

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
2022-02-25 09:14:17 +01:00
Gábor Farkas 5d704fd46e Loki: backend: use custom http code (#44643)
* Loki: backend: use custom http code

* simplfied comment

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>

Co-authored-by: Ivana Huckova <30407135+ivanahuckova@users.noreply.github.com>
2022-02-18 14:35:39 +01:00
Torkel Ödegaard a757a57c6e Prometheus: Show executed query in inspector (#45362)
* Prometheus: Show executed query in inspector

* Loki: Improving test
2022-02-15 15:02:03 +01:00
Gábor Farkas 96ae44ee2c loki: alerting: do not round timestamps to seconds (#45057) 2022-02-10 12:53:46 +01:00
Gábor Farkas d9d1f8520e loki: backend-queries: set interval-value in dataframe (#44965)
* loki: backend-queries: set interval-value in dataframe

* fixed unit test

* added interval-specific unit test
2022-02-07 13:02:04 +01:00
Gábor Farkas 3873c5e7b5 loki: alerting: better error-handling (#44700) 2022-02-01 13:34:03 +01:00
Gábor Farkas b42161a713 Loki: add from JSON to dataframes tests (#44375)
* Loki: add from JSON to dataframes tests

* fixed wrong content-type

* added -Inf test-case
2022-01-27 12:43:44 +01:00