Azure: Basic Logs support (#88025)

* Azure monitor: Basic Logs frontend (#85905) * adds datasource level config for enabling basic logs * add basiclogsquery type to query json * add toggle between basic and analytics * adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected * add check to remove UI if alerting * tests for logsmanagement component * tests for logs query editor * tests for time mangement control * remove unused imports * clears query whenever toggle changes from basic <-> analytics * add test to account for clearning query * Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx wording Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * update dependency list * clear basic logs if resources change * fix tests --------- Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Azure Monitor: Basic Logs modal acknowledgement (#86244) * adds datasource level config for enabling basic logs * add basiclogsquery type to query json * add toggle between basic and analytics * adds basic logs toggle from UI, blocks time picker to only dashboard if basic logs is selected * add check to remove UI if alerting * tests for logsmanagement component * tests for logs query editor * tests for time mangement control * remove unused imports * add confirm modal * clears query whenever toggle changes from basic <-> analytics * add test to account for clearning query * adds modal acknowledgement for basic logs query * tests for handling modal logic * basic logs ack type * Update public/app/plugins/datasource/azuremonitor/components/ConfigEditor/BasicLogsToggle.tsx wording Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsQueryEditor.tsx spelling Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * update dependency list * clear basic logs if resources change * remove modal from config page * remove basic logs query ack type * add modal acknowledgement to toggle between basic and analytics * clear query if resources change * fix tests * fix tests * Update public/app/plugins/datasource/azuremonitor/components/LogsQueryEditor/LogsManagement.tsx Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * fix tests --------- Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Azure Monitor: Basic Logs Backend (#87653) * fix logic for showingBasicLogsToggle * move to utils function and add basiclogsquery in apply template variable * add backend safeguards for basiclogsqueries * adds support for calling search or query apis based on whether it is basic logs or not * add tests for utils * initial test for basic logs query in the backend * tests for basic logs * remve comment * simplify checks for basic logs * adds fromAlert prop for azure monitor backend services * adds fromAlert check fo basic logs * fix working and empty tags * add telemetry for basic logs * remove import from grafana core package * change fromAlert true in tests * change the way tests catch errors * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/utils.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * restructure code to only run basic logs checks if basiclogsflag is true * data retention warning * tests for calculate time range * Simplify determining if request is from alerting * Fix lint and bool check * Fix tests * clarify data retention --------- Co-authored-by: Jocelyn <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com> * Azure Monitor: Basic Logs data volume notification (#88009) * frontend changes for data ingested warning * initial logic for getResource * payload processing * create basicLogs usage function * add utils for converting time and getting the data volume query for basic logs * frontend updates for showing the data ingested for the given query * frontend tests * add check for when no dataIngested is returned * remove backend.logger prints * comment on what function does * fix merge * make resource URI regex case insensitive * add support for workspace variables in basic logs flow * add undefined check * structure and add tests for variable support * Update pkg/tsdb/azuremonitor/loganalytics/azure-log-analytics-datasource.go Co-authored-by: Andreas Christou <andreas.christou@grafana.com> * add tracing for basic logs usage request * clean up data volume query struct * use async/await instead of callback * fix parameters for getApiURL * restrict time on usage query to 8 days max * add time to dependency array to refetch basic logs usage * move time check implementation to backend * fix utils tests --------- Co-authored-by: Jocelyn <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com> --------- Co-authored-by: jcolladokuri <jcolladokuri@microsoft.com> Co-authored-by: jcolladokuri <jocelyncollado52@gmail.com>
2024-05-28 18:06:27 +01:00
parent 0b2fab9967
commit 3eea71cc6b
35 changed files with 1188 additions and 55 deletions
@@ -3,7 +3,9 @@ package loganalytics
 import (
 	"fmt"
 	"regexp"
+	"strconv"
 	"strings"
+	"time"

 	"github.com/grafana/grafana-plugin-sdk-go/data"
 	"github.com/grafana/grafana/pkg/tsdb/azuremonitor/kinds/dataquery"
@@ -37,6 +39,25 @@ func AddConfigLinks(frame data.Frame, dl string, title *string) data.Frame {
 	return frame
 }

+// Check whether a query should be handled as basic logs query
+// 2. resource selected is a workspace
+// 3. query is not an alerts query
+// 4. number of selected resources is exactly one
+func meetsBasicLogsCriteria(resources []string, fromAlert bool) (bool, error) {
+	if fromAlert {
+		return false, fmt.Errorf("basic Logs queries cannot be used for alerts")
+	}
+	if len(resources) != 1 {
+		return false, fmt.Errorf("basic logs queries cannot be run against multiple resources")
+	}
+
+	if !strings.Contains(strings.ToLower(resources[0]), "microsoft.operationalinsights/workspaces") {
+		return false, fmt.Errorf("basic Logs queries may only be run against Log Analytics workspaces")
+	}
+
+	return true, nil
+}
+
 func ParseResultFormat(queryResultFormat *dataquery.ResultFormat, queryType dataquery.AzureQueryType) dataquery.ResultFormat {
 	var resultFormat dataquery.ResultFormat
 	if queryResultFormat != nil {
@@ -55,17 +76,22 @@ func ParseResultFormat(queryResultFormat *dataquery.ResultFormat, queryType data
 	return resultFormat
 }

-func getApiURL(resourceOrWorkspace string, isAppInsightsQuery bool) string {
+func getApiURL(resourceOrWorkspace string, isAppInsightsQuery bool, basicLogsQuery bool) string {
 	matchesResourceURI, _ := regexp.MatchString("^/subscriptions/", resourceOrWorkspace)

+	queryOrSearch := "query"
+	if basicLogsQuery {
+		queryOrSearch = "search"
+	}
+
 	if matchesResourceURI {
 		if isAppInsightsQuery {
 			componentName := resourceOrWorkspace[strings.LastIndex(resourceOrWorkspace, "/")+1:]
 			return fmt.Sprintf("v1/apps/%s/query", componentName)
 		}
-		return fmt.Sprintf("v1%s/query", resourceOrWorkspace)
+		return fmt.Sprintf("v1%s/%s", resourceOrWorkspace, queryOrSearch)
 	} else {
-		return fmt.Sprintf("v1/workspaces/%s/query", resourceOrWorkspace)
+		return fmt.Sprintf("v1/workspaces/%s/%s", resourceOrWorkspace, queryOrSearch)
 	}
 }

@@ -88,3 +114,21 @@ func retrieveResources(query dataquery.AzureLogsQuery) ([]string, string) {

 	return resources, resourceOrWorkspace
 }
+
+func ConvertTime(timeStamp string) (time.Time, error) {
+	// Convert the timestamp string to an int64
+	timestampInt, err := strconv.ParseInt(timeStamp, 10, 64)
+	if err != nil {
+		// Handle error
+		return time.Time{}, err
+	}
+
+	// Convert the Unix timestamp (in milliseconds) to a time.Time
+	convTimeStamp := time.Unix(0, timestampInt*int64(time.Millisecond))
+
+	return convTimeStamp, nil
+}
+
+func GetDataVolumeRawQuery(table string) string {
+	return fmt.Sprintf("Usage \n| where DataType == \"%s\"\n| where IsBillable == true\n| summarize BillableDataGB = round(sum(Quantity) / 1000, 3)", table)
+}