Azure: Basic Logs support (#88025)
* Azure monitor: Basic Logs frontend (#85905) * adds datasource level config for enabling basic logs * add basiclogsquery type to query json * add toggle between basic and analytics * adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected * add check to remove UI if alerting * tests for logsmanagement component * tests for logs query editor * tests for time mangement control * remove unused imports * clears query whenever toggle changes from basic <-> analytics * add test to account for clearning query * Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx wording Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * update dependency list * clear basic logs if resources change * fix tests --------- Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Azure Monitor: Basic Logs modal acknowledgement (#86244) * adds datasource level config for enabling basic logs * add basiclogsquery type to query json * add toggle between basic and analytics * adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected * add check to remove UI if alerting * tests for logsmanagement component * tests for logs query editor * tests for time mangement control * remove unused imports * add confirm modal * clears query whenever toggle changes from basic <-> analytics * add test to account for clearning query * adds modal acknowledgement for basic logs query * tests for handling modal logic * basic logs ack type * Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx wording Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * update dependency list * clear basic logs if resources change * remove modal from config page * remove basic logs query ack type * add modal acknowledgement to toggle between basic and analytics * clear query if resources change * fix tests * fix tests * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsManagement.tsx Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * fix tests --------- Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Azure Monitor: Basic Logs Backend (#87653) * fix logic for showingBasicLogsToggle * move to utils function and add basiclogsquery in apply template variable * add backend safeguards for basiclogsqueries * adds support for calling search or query apis based on whether it is basic logs or not * add tests for utils * initial test for basic logs query in the backend * tests for basic logs * remve comment * simplify checks for basic logs * adds fromAlert prop for azure monitor backend services * adds fromAlert check fo basic logs * fix working and empty tags * add telemetry for basic logs * remove import from grafana core package * change fromAlert true in tests * change the way tests catch errors * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * restructure code to only run basic logs checks if basiclogsflag is true * data retention warning * tests for calculate time range * Simplify determining if request is from alerting * Fix lint and bool check * Fix tests * clarify data retention --------- Co-authored-by: Jocelyn <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com> * Azure Monitor: Basic Logs data volume notification (#88009) * frontend changes for data ingested warning * initial logic for getResource * payload processing * create basicLogs usage function * add utils for converting time and getting the data volume query for basic logs * frontend updates for showing the data ingested for the given query * frontend tests * add check for when no dataIngested is returned * remove backend.logger prints * comment on what function does * fix merge * make resource URI regex case insensitive * add support for workspace variables in basic logs flow * add undefined check * structure and add tests for variable support * Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * add tracing for basic logs usage request * clean up data volume query struct * use async/await instead of callback * fix parameters for getApiURL * restrict time on usage query to 8 days max * add time to dependency array to refetch basic logs usage * move time check implementation to backend * fix utils tests --------- Co-authored-by: Jocelyn <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com> --------- Co-authored-by: jcolladokuri <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com>
This commit is contained in:
Andreas Christou
co-authored by
Jocelyn
jcolladokuri
Jocelyn
jcolladokuri
parent
0b2fab9967
commit
3eea71cc6b
@@ -27,16 +27,116 @@ import (
|
||||
)
|
||||
|
||||
func (e *AzureLogAnalyticsDatasource) ResourceRequest(rw http.ResponseWriter, req *http.Request, cli *http.Client) (http.ResponseWriter, error) {
|
||||
if req.URL.Path == "/usage/basiclogs" {
|
||||
newUrl := &url.URL{
|
||||
Scheme: req.URL.Scheme,
|
||||
Host: req.URL.Host,
|
||||
Path: "/v1/query",
|
||||
}
|
||||
return e.GetBasicLogsUsage(req.Context(), newUrl.String(), cli, rw, req.Body)
|
||||
}
|
||||
return e.Proxy.Do(rw, req, cli)
|
||||
}
|
||||
|
||||
// builds and executes a new query request that will get the data ingeted for the given table in the basic logs query
|
||||
func (e *AzureLogAnalyticsDatasource) GetBasicLogsUsage(ctx context.Context, url string, client *http.Client, rw http.ResponseWriter, reqBody io.ReadCloser) (http.ResponseWriter, error) {
|
||||
// read the full body
|
||||
originalPayload, readErr := io.ReadAll(reqBody)
|
||||
if readErr != nil {
|
||||
return rw, fmt.Errorf("failed to read request body %w", readErr)
|
||||
}
|
||||
var payload BasicLogsUsagePayload
|
||||
jsonErr := json.Unmarshal(originalPayload, &payload)
|
||||
if jsonErr != nil {
|
||||
return rw, fmt.Errorf("error decoding basic logs table usage payload: %w", jsonErr)
|
||||
}
|
||||
table := payload.Table
|
||||
|
||||
from, fromErr := ConvertTime(payload.From)
|
||||
if fromErr != nil {
|
||||
return rw, fmt.Errorf("failed to convert from time: %w", fromErr)
|
||||
}
|
||||
|
||||
to, toErr := ConvertTime(payload.To)
|
||||
if toErr != nil {
|
||||
return rw, fmt.Errorf("failed to convert to time: %w", toErr)
|
||||
}
|
||||
|
||||
// basic logs queries only show data for last 8 days or less
|
||||
// data volume query should also only calculate volume for last 8 days if time range exceeds that.
|
||||
diff := to.Sub(from).Hours()
|
||||
if diff > float64(MaxHoursBasicLogs) {
|
||||
from = to.Add(-time.Duration(MaxHoursBasicLogs) * time.Hour)
|
||||
}
|
||||
|
||||
dataVolumeQueryRaw := GetDataVolumeRawQuery(table)
|
||||
dataVolumeQuery := &AzureLogAnalyticsQuery{
|
||||
Query: dataVolumeQueryRaw,
|
||||
DashboardTime: true, // necessary to ensure TimeRange property is used since query will not have an in-query time filter
|
||||
TimeRange: backend.TimeRange{
|
||||
From: from,
|
||||
To: to,
|
||||
},
|
||||
TimeColumn: "TimeGenerated",
|
||||
Resources: []string{payload.Resource},
|
||||
QueryType: dataquery.AzureQueryTypeAzureLogAnalytics,
|
||||
URL: getApiURL(payload.Resource, false, false),
|
||||
}
|
||||
|
||||
req, err := e.createRequest(ctx, url, dataVolumeQuery)
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
|
||||
_, span := tracing.DefaultTracer().Start(ctx, "azure basic logs usage query", trace.WithAttributes(
|
||||
attribute.String("target", dataVolumeQuery.Query),
|
||||
attribute.String("table", table),
|
||||
attribute.Int64("from", dataVolumeQuery.TimeRange.From.UnixNano()/int64(time.Millisecond)),
|
||||
attribute.Int64("until", dataVolumeQuery.TimeRange.To.UnixNano()/int64(time.Millisecond)),
|
||||
))
|
||||
defer span.End()
|
||||
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
|
||||
defer func() {
|
||||
if err := resp.Body.Close(); err != nil {
|
||||
e.Logger.Warn("Failed to close response body for data volume request", "err", err)
|
||||
}
|
||||
}()
|
||||
|
||||
logResponse, err := e.unmarshalResponse(resp)
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
|
||||
t, err := logResponse.GetPrimaryResultTable()
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
|
||||
num := t.Rows[0][0].(json.Number)
|
||||
value, err := num.Float64()
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
_, err = rw.Write([]byte(fmt.Sprintf("%f", value)))
|
||||
if err != nil {
|
||||
return rw, err
|
||||
}
|
||||
|
||||
return rw, err
|
||||
}
|
||||
|
||||
// executeTimeSeriesQuery does the following:
|
||||
// 1. build the AzureMonitor url and querystring for each query
|
||||
// 2. executes each query by calling the Azure Monitor API
|
||||
// 3. parses the responses for each query into data frames
|
||||
func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string) (*backend.QueryDataResponse, error) {
|
||||
func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context, originalQueries []backend.DataQuery, dsInfo types.DatasourceInfo, client *http.Client, url string, fromAlert bool) (*backend.QueryDataResponse, error) {
|
||||
result := backend.NewQueryDataResponse()
|
||||
queries, err := e.buildQueries(ctx, originalQueries, dsInfo)
|
||||
queries, err := e.buildQueries(ctx, originalQueries, dsInfo, fromAlert)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -53,23 +153,38 @@ func (e *AzureLogAnalyticsDatasource) ExecuteTimeSeriesQuery(ctx context.Context
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo, appInsightsRegExp *regexp.Regexp) (*AzureLogAnalyticsQuery, error) {
|
||||
func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo, appInsightsRegExp *regexp.Regexp, fromAlert bool) (*AzureLogAnalyticsQuery, error) {
|
||||
queryJSONModel := types.LogJSONQuery{}
|
||||
err := json.Unmarshal(query.JSON, &queryJSONModel)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to decode the Azure Log Analytics query object from JSON: %w", err)
|
||||
}
|
||||
|
||||
var queryString string
|
||||
appInsightsQuery := false
|
||||
dashboardTime := false
|
||||
timeColumn := ""
|
||||
azureLogAnalyticsTarget := queryJSONModel.AzureLogAnalytics
|
||||
basicLogsQuery := false
|
||||
|
||||
resultFormat := ParseResultFormat(azureLogAnalyticsTarget.ResultFormat, dataquery.AzureQueryTypeAzureLogAnalytics)
|
||||
|
||||
basicLogsQueryFlag := false
|
||||
if azureLogAnalyticsTarget.BasicLogsQuery != nil {
|
||||
basicLogsQueryFlag = *azureLogAnalyticsTarget.BasicLogsQuery
|
||||
}
|
||||
|
||||
resources, resourceOrWorkspace := retrieveResources(azureLogAnalyticsTarget)
|
||||
appInsightsQuery = appInsightsRegExp.Match([]byte(resourceOrWorkspace))
|
||||
|
||||
if basicLogsQueryFlag {
|
||||
if meetsBasicLogsCriteria, meetsBasicLogsCriteriaErr := meetsBasicLogsCriteria(resources, fromAlert); meetsBasicLogsCriteriaErr != nil {
|
||||
return nil, meetsBasicLogsCriteriaErr
|
||||
} else {
|
||||
basicLogsQuery = meetsBasicLogsCriteria
|
||||
}
|
||||
}
|
||||
|
||||
if azureLogAnalyticsTarget.Query != nil {
|
||||
queryString = *azureLogAnalyticsTarget.Query
|
||||
}
|
||||
@@ -86,7 +201,7 @@ func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo
|
||||
}
|
||||
}
|
||||
|
||||
apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery)
|
||||
apiURL := getApiURL(resourceOrWorkspace, appInsightsQuery, basicLogsQuery)
|
||||
|
||||
rawQuery, err := macros.KqlInterpolate(query, dsInfo, queryString, "TimeGenerated")
|
||||
if err != nil {
|
||||
@@ -105,10 +220,11 @@ func buildLogAnalyticsQuery(query backend.DataQuery, dsInfo types.DatasourceInfo
|
||||
AppInsightsQuery: appInsightsQuery,
|
||||
DashboardTime: dashboardTime,
|
||||
TimeColumn: timeColumn,
|
||||
BasicLogs: basicLogsQuery,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries []backend.DataQuery, dsInfo types.DatasourceInfo) ([]*AzureLogAnalyticsQuery, error) {
|
||||
func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries []backend.DataQuery, dsInfo types.DatasourceInfo, fromAlert bool) ([]*AzureLogAnalyticsQuery, error) {
|
||||
azureLogAnalyticsQueries := []*AzureLogAnalyticsQuery{}
|
||||
appInsightsRegExp, err := regexp.Compile("providers/Microsoft.Insights/components")
|
||||
if err != nil {
|
||||
@@ -117,7 +233,7 @@ func (e *AzureLogAnalyticsDatasource) buildQueries(ctx context.Context, queries
|
||||
|
||||
for _, query := range queries {
|
||||
if query.QueryType == string(dataquery.AzureQueryTypeAzureLogAnalytics) {
|
||||
azureLogAnalyticsQuery, err := buildLogAnalyticsQuery(query, dsInfo, appInsightsRegExp)
|
||||
azureLogAnalyticsQuery, err := buildLogAnalyticsQuery(query, dsInfo, appInsightsRegExp, fromAlert)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to build azure log analytics query: %w", err)
|
||||
}
|
||||
@@ -161,6 +277,7 @@ func (e *AzureLogAnalyticsDatasource) executeQuery(ctx context.Context, query *A
|
||||
|
||||
_, span := tracing.DefaultTracer().Start(ctx, "azure log analytics query", trace.WithAttributes(
|
||||
attribute.String("target", query.Query),
|
||||
attribute.Bool("basic_logs", query.BasicLogs),
|
||||
attribute.Int64("from", query.TimeRange.From.UnixNano()/int64(time.Millisecond)),
|
||||
attribute.Int64("until", query.TimeRange.To.UnixNano()/int64(time.Millisecond)),
|
||||
attribute.Int64("datasource_id", dsInfo.DatasourceID),
|
||||
|
||||
Reference in New Issue
Block a user