CI Migration: Backport pkg/build (daggerbuild) and GHAs to release-12.0.3 (#107997)
* CI: move grafana-build into pkg/build (#105640) * move grafana-build into pkg/build * Update go.mod/go.sum files after cherry-pick build - Updates from workspace sync after cherry-picking daggerbuild package - Resolves dependency conflicts from pkg/build integration - Required for successful build after CI migration backport * CI Migration: Replace .github with main branch version - Complete replacement of .github directory from main branch - Includes updated workflows, actions, and configuration files - Ensures all CI improvements from main are included - Next step: fix branch-specific issues in separate commits * Fix branch triggers: Update workflows to target release branches - Update 8 workflows to trigger on release-* instead of main - Ensures workflows run on pull requests to release branches - Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator * Fix branch triggers: Update workflows to target both main and release branches - CORRECTED: Include both main and release-* branch patterns - Ensures workflows run on both main branch (original functionality) and release branches (needed for backport) - Previous commit incorrectly removed main branch support - Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator * CI Migration: Update .gitignore to allow OSS wire file to be committed - Remove **/wire_gen.go (allow pkg/server/wire_gen.go to be committed) - Add /pkg/server/enterprise_wire_gen.go (keep enterprise wire file ignored) - This enables the new committed wire files architecture for CI * CI Migration: Add enhanced wire tool with automatic build constraints - Backport enhanced wire tool from main branch with -gen_tags flag - This enables automatic regeneration with proper constraints to prevent conflicts - Wire file contains all required functions: Initialize, InitializeDocumentBuilders, etc. - Solves the missing wire functions issue causing CI failures - Enhanced tool ensures build constraints persist on regeneration * CI Migration: Update Makefile to use enhanced wire tool with build constraints - Update gen-go target to use -gen_tags flag for automatic build constraints - Ensures future engineers get proper build constraints when running 'make gen-go' - Matches main branch approach for consistent wire file generation - Removes dependency on WIRE_TAGS variable and GO_RACE_FLAG for simplicity * Add gen-enterprise-go target to Makefile for release-12.0.3 - Backports gen-enterprise-go target from main branch - Enables enterprise wire file generation in release branches - Part of CI migration backport wire infrastructure fixes - Resolves CI issues where enterprise wire functions were missing * Fix wire tool golden file format for Drone CI compatibility - Update all 43 golden test files to match main branch format - Change go:generate command from github.com/google/wire/cmd/wire to ./pkg/build/wire/cmd/wire/main.go - Remove legacy '// +build !wireinject' constraints - Fixes Drone CI test failures that were blocking OSS PR merge - All wire tests now pass successfully This aligns the backport branch with the golden file format updates that were made in main branch when the wire tool was enhanced. * Update swagger specs to include enterprise APIs - Regenerated api-enterprise-spec.json with enterprise API definitions - Updated api-merged.json with merged OSS and enterprise specs - Regenerated openapi3.json with complete API specifications - Fixes enterprise CI swagger generation validation failures - Enterprise APIs now properly detected and documented * Fix OSS integration tests missing enterprise build tags in Drone CI - Add -tags=enterprise to all integration test steps in lib.star - Fixes test-backend-integration, postgres-integration-tests, and mysql-integration-tests - Regenerate .drone.yml with proper enterprise build tags - Resolves 'server could not find the requested resource' errors for enterprise APIs - Ensures enterprise APIs (querylibrary, reporting, banners, scim) are registered during tests - Aligns OSS Drone CI with GitHub Actions behavior and enterprise repository * Fix CODEOWNERS for release-12.0.3 compatibility - Remove 21 entries that reference files/directories not present in release branch - Resolves File Exist Checker failures in codeowners-validator workflow - Maintains team ownership assignments from main branch for existing files - Lines removed: .air.toml, apps/secret/, apps/iam/, e2e-playwright/, packages/grafana-alerting/, etc. * baldm0mma/ update releasefinder * Add i18n-extract script for release branch compatibility - Add 'i18n-extract': 'make i18n-extract' to package.json scripts - Resolves i18n-verify workflow failure that expects this script - The make target already exists and handles OSS + enterprise i18n extraction - Maintains i18n verification functionality from main branch workflows - Simple safe addition that calls existing make infrastructure * Phase 4: Fix GitHub Actions workflow branch triggers - Add release-*.*.* pattern to 8 workflows missing it - Fix duplicate release branch patterns in 3 workflows - Ensure consistent branch trigger format across all workflows - Critical workflows now properly trigger on release branches Fixes workflows: actionlint, backend-code-checks, go-lint, reject-gh-secrets, run-schema-v2-e2e, shellcheck, swagger-gen, trivy-scan * Fix: Backport E2E runner infrastructure for Enterprise CI - Add e2e/main.go and e2e/internal/ directory from main branch - Add urfave/cli/v3 dependency required by E2E runner - Fixes Enterprise CI failure: 'Build E2E test runner' and 'Build & package Grafana for e2e tests' - Root cause: E2E runner infrastructure was added to main after release-12.0.3 branch creation - Solution: Backport missing E2E runner files to enable Enterprise CI completion Resolves: no Go files in /opt/actions-runner/_work/grafana-enterprise/grafana-enterprise/grafana/e2e * Fix: Update go.mod dependency classification for urfave/cli/v3 - Change urfave/cli/v3 from indirect to direct dependency - Fixes Go Workspace Check failure in CI - Required after adding E2E runner infrastructure that directly imports urfave/cli/v3 - Resolves: make update-workspace corrects dependency classification * Fix: Add team ownership for urfave/cli/v3 dependency - Assign @grafana/grafana-backend-group as owner for github.com/urfave/cli/v3@v3.3.8 - Follows existing pattern: urfave/cli v1 and v2 also owned by grafana-backend-group - Resolves Backend Code Checks / Validate Backend Configs CI failure - Required for E2E runner infrastructure dependency ownership compliance Fixes: modowners check requiring team assignment for newly added dependencies * Fix: Revert experimental E2E playwright infrastructure to stable version - Revert pr-e2e-tests.yml to stable Cypress-based E2E testing - Remove experimental storybook-verification-playwright.yml workflow - Revert run-dashboard-search-e2e.yml and release-pr.yml to use e2e/test-plugins/ - Keep stable E2E runner infrastructure (e2e/main.go + e2e/internal/) - Remove experimental playwright features per team recommendation Team feedback: Playwright tests are experimental and shouldn't be backported to stable release branches * Fix: Complete cleanup of experimental playwright dependencies - Revert package.json to stable version (remove e2e-playwright scripts and path references) - Revert playwright.config.ts to stable plugin-e2e configuration - Remove all experimental playwright infrastructure dependencies - Ensure clean stable E2E testing environment All experimental features removed per team recommendation for stable release branches * Fix: Restore working package.json configuration - Revert package.json to version 12.0.3 (working release branch version) - Fix workspace dependency resolution issues caused by incorrect revert to main branch version - FE tests, betterer, and linting should now work correctly Issue was caused by reverting package.json to main branch (12.1.0-pre) instead of keeping the working release branch configuration (12.0.3) * CI: mirror some CI dependencies (#106148) * mirror some CI dependencies * remove -v from go build * Fix: Backport missing e2e/run-suite script for daggerbuild E2E system The daggerbuild E2E system (used by OSS workflows) expects ./e2e/run-suite to exist, but we only backported the new E2E runner infrastructure. This script is needed for: - OSS workflow: 'go run ./pkg/build/e2e --suite=dashboards-suite' - Legacy Cypress configuration with video support - Integration between daggerbuild and existing Cypress test suites Resolves E2E test failures in OSS workflows where videos directory cannot be found because run-suite script was missing. * Fix: Resolve daggerbuild E2E path doubling issue - Extract just suite name from full path using filepath.Base() - Prevents doubled paths like './e2e/e2e/dashboards-suite/videos' - Resolves 'no spec files found' and 'no such file or directory' errors - GitHub Actions passes full paths like 'e2e/dashboards-suite' but run-suite script expects just 'dashboards-suite' * Infrastructure: Wholesale copy pkg/build/ from main COMPLETE DAGGERBUILD SYSTEM UPDATE: - Modern E2E system with CLI framework and --flags support - New accessibility testing system (a11y/) - New Playwright E2E testing system (e2e-playwright/) - External infrastructure improvements (musl.cc → dl.grafana.com/ci) - Updated daggerbuild core components with latest fixes - Updated Go dependencies and wire modules ARCHITECTURAL COMPATIBILITY VERIFIED: - Modern pkg/build calls: ./e2e-runner cypress --start-grafana=false --cypress-video - Our e2e runner supports: All required parameters including --suite, --env flags - GitHub Actions workflows: Pass compatible arguments - Binary builds: Successfully creates working e2e-runner RESOLVES ISSUES: - Fixes --flags parameter compatibility with GitHub Actions - Includes all follow-up infrastructure improvements - Provides complete, tested system with modern CLI framework - Eliminates external dependency failures (musl.cc timeouts) - Removes need for path doubling workarounds (modern system handles full paths correctly) Replaces incremental cherry-picking with complete, tested system from main. * Dependencies: Update Go modules after pkg/build wholesale copy DEPENDENCY UPDATES: - Updated go.work.sum with new dependencies from modern pkg/build system - Updated all workspace module dependencies (go.mod/go.sum files) - Removed pkg/build/cmd/enterprise.go (enterprise-only file, gets copied during enterprise development) ENTERPRISE INTEGRATION FIX: - enterprise.go file doesn't belong in OSS repository - Gets copied from grafana-enterprise during development mode - Main branch doesn't have this file, explaining module compatibility All Go modules now properly resolved and compatible with modern pkg/build architecture. * Dependencies: Update workspace Go module checksums after pkg/build wholesale copy - Synchronizes all go.sum files across workspace modules - Adds missing .mod.h1 entries that were required after infrastructure update - Resolves Go Workspace Check CI failures - Updates 22 modules: .citools/, apps/, pkg/ subdirectories plus go.work.sum Addresses CI error: 'Please run make update-workspace and commit the changes' * Infrastructure: Revert to pre-Playwright E2E system for release branch compatibility Strategic combination of modern daggerbuild with stable E2E infrastructure: **Modern Daggerbuild (from main):** - Latest CLI framework (github.com/urfave/cli/v3) - External infrastructure fixes (musl.cc → dl.grafana.com/ci) - Complete pkg/build/ system with all enhancements - Updated dependencies and architecture improvements **Stable E2E Infrastructure (pre-Playwright):** - pkg/build/e2e/service.go: Reverted to version before commitb6580ccb10- pkg/build/a11y/: Reverted to stable version compatible with existing infrastructure - Removed experimental pkg/build/e2e-playwright/ system - Removed experimental e2e-playwright/ test infrastructure **Why This Approach:** - GitHub Actions workflows call 'go run ./pkg/build/e2e' (modern system) - Modern system now expects only standard e2e/ directory (compatible) - Release branches have standard e2e/ infrastructure (dashboards-suite, various-suite, etc.) - Eliminates dependency on experimental Playwright infrastructure **Key Dependencies Met:** - e2e/pa11yci.conf.js: Added for a11y test compatibility - /src/e2e directory: Standard directory structure (no e2e-playwright needed) - Existing test plugins: Uses stable e2e/test-plugins/ **Result:** Best of both worlds - modern, reliable daggerbuild infrastructure with proven E2E testing that works across all release branches without requiring experimental dependencies. * Fix: Remove unused embed.go file causing golangci-lint failure - Removes pkg/build/daggerbuild/msi/embed.go with unused embed.FS variable - MSI build process uses directory mounting instead of embedded filesystem - Resolves 'var resources is unused (unused)' linting error * Fix: Update Node.js version to match working E2E configuration - Changes from v22.11.0 to v22.16.0 to match commit17952d45e4- Working commit had all E2E tests passing with same daggerbuild infrastructure - Node.js version differences can affect frontend builds and UI rendering - Ensures consistent environment between working reference and current branch * Fix: Use conditional container base for E2E compatibility - Alpine for OSS tests (better old arch dashboardScene=false compatibility) - Ubuntu for Enterprise tests (when image renderer needed) - Restores original release-12.0.3 Alpine behavior for OSS - Should resolve old arch E2E test failures * Fix: Go fmt formatting for conditional container logic * Revert: Remove unnecessary Alpine/Ubuntu conditional logic - Real issue was wrong environment variable name (dashboardScene vs DISABLE_SCENES) - Keep Ubuntu base (original main branch behavior) for simplicity - Container base was not the root cause of old arch failures * Experiment: Copy scripts/ from pre-Playwright commit (ea0ddb3fc9) - Copied scripts directory from commitea0ddb3fc9(just before Playwright migration) - This overwrites some CI migration fixes but testing if it's actually needed - Can revert back tob9b4602dbdif CI breaks Test commit to see real impact on GitHub Actions workflows. * trigger ci 01 * Fix: Copy .drone.star from main to match scripts structure - Resolves Starlark evaluation error for missing integration_test_pipelines - Aligns .drone.star with main branch scripts structure - Same fix applied to both OSS and Enterprise repositories * make drone * Fix ARM/v7 Docker build failures by using dagger run for cross-compilation - Change artifacts_cmd() in scripts/drone/steps/rgm.star to use 'dagger run go run' instead of 'go run' - Add --max-execution-steps 100000 to make drone target in Makefile to properly regenerate .drone.yml - This fixes CGO cross-compilation issues with SQLite for ARM/v7 Docker builds - Matches the approach already used successfully in Enterprise builds * Fix Docker base image references in rgm build steps - Change from symbolic names (alpine-base, ubuntu-base) to actual Docker images - alpine-base → alpine:3.21.3 - ubuntu-base → ubuntu:22.04 - Add ubuntu and alpine parameters to rgm_artifacts_step and rgm_build_docker_step functions - Fixes Docker pull errors: 'repository does not exist or may require authorization' - Aligns OSS configuration with Enterprise approach --------- Co-authored-by: Kevin Minehart <5140827+kminehart@users.noreply.github.com>
This commit is contained in:
Jev Forsberg
co-authored by
Kevin Minehart
Kevin Minehart
parent
aee6b20c7f
commit
dc6bd2a20a
@@ -2,15 +2,34 @@ package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
var httpClient = http.Client{
|
||||
Transport: &http.Transport{
|
||||
Proxy: http.ProxyFromEnvironment,
|
||||
DialContext: func(dialer *net.Dialer) func(context.Context, string, string) (net.Conn, error) {
|
||||
return dialer.DialContext
|
||||
}(&net.Dialer{
|
||||
Timeout: 30 * time.Second,
|
||||
KeepAlive: 30 * time.Second,
|
||||
}),
|
||||
ForceAttemptHTTP2: true,
|
||||
MaxIdleConns: 100,
|
||||
IdleConnTimeout: 90 * time.Second,
|
||||
TLSHandshakeTimeout: 10 * time.Second,
|
||||
ExpectContinueTimeout: 1 * time.Second,
|
||||
},
|
||||
}
|
||||
|
||||
type publisher struct {
|
||||
apiKey string
|
||||
apiURI string
|
||||
@@ -264,7 +283,7 @@ func (p *publisher) postRequest(url string, obj any, desc string) error {
|
||||
req.Header.Add("Authorization", "Bearer "+p.apiKey)
|
||||
req.Header.Add("Content-Type", "application/json")
|
||||
|
||||
res, err := http.DefaultClient.Do(req)
|
||||
res, err := httpClient.Do(req)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
@@ -55,7 +55,7 @@ while IFS=" " read -r -a package; do
|
||||
# (non-zero if any of the packages failed the checks)
|
||||
if [ "$STATUS" -gt 0 ]; then
|
||||
EXIT_CODE=1
|
||||
GITHUB_MESSAGE="${GITHUB_MESSAGE}**\\\`${PACKAGE_PATH}\\\`** has possible breaking changes<br />"
|
||||
GITHUB_MESSAGE="${GITHUB_MESSAGE}**<code>${PACKAGE_PATH}</code>** has possible breaking changes<br />"
|
||||
GITHUB_LEVITATE_MARKDOWN+="<h3>${PACKAGE_PATH}</h3>${CURRENT_REPORT}<br>"
|
||||
fi
|
||||
|
||||
@@ -67,5 +67,10 @@ echo "message=$GITHUB_MESSAGE" >>"$GITHUB_OUTPUT"
|
||||
mkdir -p ./levitate
|
||||
echo "$GITHUB_LEVITATE_MARKDOWN" >./levitate/levitate.md
|
||||
|
||||
if [[ "$IS_FORK" == "true" ]]; then
|
||||
cat ./levitate/levitate.md >> "$GITHUB_STEP_SUMMARY"
|
||||
exit $EXIT_CODE
|
||||
fi
|
||||
|
||||
# We will exit the workflow accordingly at another step
|
||||
exit 0
|
||||
|
||||
@@ -44,7 +44,7 @@ done
|
||||
shift $((OPTIND - 1))
|
||||
|
||||
if [[ ${#dirs[@]} -eq 0 ]]; then
|
||||
dirs+=("./...")
|
||||
readarray -t dirs <<< "$(find . -type f -name 'go.mod' -exec dirname '{}' ';' | awk '{ print $1 "/..."; }')"
|
||||
fi
|
||||
if [ -z "$beginningWith" ]; then
|
||||
for pkg in "${dirs[@]}"; do
|
||||
|
||||
@@ -100,7 +100,7 @@ if [[ $n -gt $m ]]; then
|
||||
exit 1
|
||||
fi
|
||||
if [[ ${#dirs[@]} -eq 0 ]]; then
|
||||
dirs+=("./...")
|
||||
readarray -t dirs <<< "$(find . -type f -name 'go.mod' -exec dirname '{}' ';' | awk '{ print $1 "/..."; }')"
|
||||
fi
|
||||
# If dirs is just ("-"), read from stdin instead.
|
||||
if [[ ${#dirs[@]} -eq 1 && "${dirs[0]}" == "-" ]]; then
|
||||
|
||||
@@ -0,0 +1,9 @@
|
||||
"""
|
||||
Utilities / functions for working with dagger pipelines
|
||||
"""
|
||||
|
||||
def with_dagger_install(commands = [], dagger_version = ""):
|
||||
return [
|
||||
"wget -qO- https://github.com/dagger/dagger/releases/download/{}/dagger_{}_linux_amd64.tar.gz | tar zx -C /bin".format(dagger_version, dagger_version),
|
||||
"apk add docker",
|
||||
] + commands
|
||||
@@ -11,26 +11,10 @@ load(
|
||||
"docs_pipelines",
|
||||
"trigger_docs_main",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/integration_tests.star",
|
||||
"integration_tests",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/lint_backend.star",
|
||||
"lint_backend_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_backend.star",
|
||||
"test_backend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/trigger_downstream.star",
|
||||
"enterprise_downstream_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/verify_storybook.star",
|
||||
"verify_storybook",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/utils/utils.star",
|
||||
"failure_template",
|
||||
@@ -60,20 +44,14 @@ def main_pipelines():
|
||||
# Let's make an effort to reduce the amount of string constants in "depends_on" lists.
|
||||
pipelines = [
|
||||
docs_pipelines(ver_mode, trigger_docs_main()),
|
||||
test_backend(trigger, ver_mode),
|
||||
lint_backend_pipeline(trigger, ver_mode),
|
||||
verify_storybook(trigger, ver_mode),
|
||||
build_e2e(trigger, ver_mode),
|
||||
integration_tests(trigger, prefix = ver_mode, ver_mode = ver_mode),
|
||||
enterprise_downstream_pipeline(),
|
||||
notify_pipeline(
|
||||
name = "main-notify",
|
||||
slack_channel = "grafana-ci-notifications",
|
||||
trigger = dict(trigger, status = ["failure"]),
|
||||
depends_on = [
|
||||
"main-test-backend",
|
||||
"main-build-e2e-publish",
|
||||
"main-integration-tests",
|
||||
],
|
||||
template = failure_template,
|
||||
secret = "slack_webhook",
|
||||
|
||||
@@ -3,10 +3,6 @@ This module returns all pipelines used in the event of a pull request.
|
||||
It also includes a function generating a PR trigger from a list of included and excluded paths.
|
||||
"""
|
||||
|
||||
load(
|
||||
"scripts/drone/pipelines/benchmarks.star",
|
||||
"integration_benchmarks",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/build.star",
|
||||
"build_e2e",
|
||||
@@ -16,26 +12,6 @@ load(
|
||||
"docs_pipelines",
|
||||
"trigger_docs_pr",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/integration_tests.star",
|
||||
"integration_tests",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/lint_backend.star",
|
||||
"lint_backend_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/shellcheck.star",
|
||||
"shellcheck_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/swagger_gen.star",
|
||||
"swagger_gen",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_backend.star",
|
||||
"test_backend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/verify_drone.star",
|
||||
"verify_drone",
|
||||
@@ -44,10 +20,6 @@ load(
|
||||
"scripts/drone/pipelines/verify_starlark.star",
|
||||
"verify_starlark",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/verify_storybook.star",
|
||||
"verify_storybook",
|
||||
)
|
||||
|
||||
ver_mode = "pr"
|
||||
trigger = {
|
||||
@@ -77,72 +49,8 @@ def pr_pipelines():
|
||||
),
|
||||
ver_mode,
|
||||
),
|
||||
verify_storybook(
|
||||
get_pr_trigger(
|
||||
include_paths = ["packages/grafana-ui/**"],
|
||||
),
|
||||
ver_mode,
|
||||
),
|
||||
test_backend(
|
||||
get_pr_trigger(
|
||||
include_paths = [
|
||||
"Makefile",
|
||||
"pkg/**",
|
||||
"packaging/**",
|
||||
".drone.yml",
|
||||
"conf/**",
|
||||
"go.sum",
|
||||
"go.mod",
|
||||
"public/app/plugins/**/plugin.json",
|
||||
"docs/sources/setup-grafana/configure-grafana/feature-toggles/**",
|
||||
"devenv/**",
|
||||
"apps/**",
|
||||
],
|
||||
),
|
||||
ver_mode,
|
||||
),
|
||||
lint_backend_pipeline(
|
||||
get_pr_trigger(
|
||||
include_paths = [
|
||||
".golangci.toml",
|
||||
"Makefile",
|
||||
"pkg/**",
|
||||
"packaging/**",
|
||||
".drone.yml",
|
||||
"conf/**",
|
||||
"go.sum",
|
||||
"go.mod",
|
||||
"public/app/plugins/**/plugin.json",
|
||||
"devenv/**",
|
||||
".bingo/**",
|
||||
"apps/**",
|
||||
],
|
||||
),
|
||||
ver_mode,
|
||||
),
|
||||
build_e2e(trigger, ver_mode),
|
||||
integration_tests(
|
||||
get_pr_trigger(
|
||||
include_paths = [
|
||||
"pkg/**",
|
||||
"packaging/**",
|
||||
".drone.yml",
|
||||
"conf/**",
|
||||
"go.sum",
|
||||
"go.mod",
|
||||
"public/app/plugins/**/plugin.json",
|
||||
],
|
||||
),
|
||||
prefix = ver_mode,
|
||||
),
|
||||
docs_pipelines(ver_mode, trigger_docs_pr()),
|
||||
shellcheck_pipeline(),
|
||||
swagger_gen(
|
||||
ver_mode,
|
||||
),
|
||||
integration_benchmarks(
|
||||
prefix = ver_mode,
|
||||
),
|
||||
]
|
||||
|
||||
def get_pr_trigger(include_paths = None, exclude_paths = None):
|
||||
|
||||
@@ -2,11 +2,6 @@
|
||||
This module returns all the pipelines used in the event of a release along with supporting functions.
|
||||
"""
|
||||
|
||||
load(
|
||||
"scripts/drone/services/services.star",
|
||||
"integration_test_services",
|
||||
"integration_test_services_volumes",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/steps/github.star",
|
||||
"github_app_generate_token_step",
|
||||
@@ -16,19 +11,9 @@ load(
|
||||
load(
|
||||
"scripts/drone/steps/lib.star",
|
||||
"compile_build_cmd",
|
||||
"download_grabpl_step",
|
||||
"identify_runner_step",
|
||||
"memcached_integration_tests_steps",
|
||||
"mysql_integration_tests_steps",
|
||||
"postgres_integration_tests_steps",
|
||||
"publish_grafanacom_step",
|
||||
"publish_linux_packages_step",
|
||||
"redis_integration_tests_steps",
|
||||
"remote_alertmanager_integration_tests_steps",
|
||||
"verify_gen_cue_step",
|
||||
"verify_gen_jsonnet_step",
|
||||
"verify_grafanacom_step",
|
||||
"wire_install_step",
|
||||
"yarn_install_step",
|
||||
)
|
||||
load(
|
||||
@@ -255,47 +240,6 @@ def publish_npm_pipelines():
|
||||
),
|
||||
]
|
||||
|
||||
def integration_test_pipelines():
|
||||
"""
|
||||
Trigger integration tests on release builds
|
||||
|
||||
These pipelines should be triggered when we have a release that does a lot of
|
||||
cherry-picking and we still want to have all the integration tests run on that
|
||||
particular build.
|
||||
|
||||
Returns:
|
||||
List of Drone pipelines
|
||||
"""
|
||||
trigger = {
|
||||
"event": ["promote"],
|
||||
"target": "integration-tests",
|
||||
}
|
||||
pipelines = []
|
||||
volumes = integration_test_services_volumes()
|
||||
integration_test_steps = postgres_integration_tests_steps() + \
|
||||
mysql_integration_tests_steps("mysql80", "8.0") + \
|
||||
redis_integration_tests_steps() + \
|
||||
memcached_integration_tests_steps() + \
|
||||
remote_alertmanager_integration_tests_steps()
|
||||
|
||||
pipelines.append(pipeline(
|
||||
name = "integration-tests",
|
||||
trigger = trigger,
|
||||
services = integration_test_services(),
|
||||
steps = [
|
||||
download_grabpl_step(),
|
||||
identify_runner_step(),
|
||||
verify_gen_cue_step(),
|
||||
verify_gen_jsonnet_step(),
|
||||
wire_install_step(),
|
||||
] +
|
||||
integration_test_steps,
|
||||
environment = {"EDITION": "oss"},
|
||||
volumes = volumes,
|
||||
))
|
||||
|
||||
return pipelines
|
||||
|
||||
def verify_release_pipeline(
|
||||
name = "verify-prerelease-assets",
|
||||
bucket = from_secret(prerelease_bucket),
|
||||
|
||||
@@ -2,26 +2,6 @@
|
||||
This module returns all the pipelines used in the event of pushes to an RRC branch.
|
||||
"""
|
||||
|
||||
load(
|
||||
"scripts/drone/pipelines/integration_tests.star",
|
||||
"integration_tests",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/lint_backend.star",
|
||||
"lint_backend_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/lint_frontend.star",
|
||||
"lint_frontend_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_backend.star",
|
||||
"test_backend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_frontend.star",
|
||||
"test_frontend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/steps/lib.star",
|
||||
"enterprise_downstream_step",
|
||||
@@ -48,11 +28,6 @@ trigger = {
|
||||
|
||||
def rrc_patch_pipelines():
|
||||
pipelines = [
|
||||
test_frontend(trigger, ver_mode),
|
||||
lint_frontend_pipeline(trigger, ver_mode),
|
||||
test_backend(trigger, ver_mode),
|
||||
lint_backend_pipeline(trigger, ver_mode),
|
||||
integration_tests(trigger, prefix = ver_mode, ver_mode = ver_mode),
|
||||
rrc_enterprise_downstream_pipeline(trigger = trigger),
|
||||
]
|
||||
|
||||
@@ -68,6 +43,5 @@ def rrc_enterprise_downstream_pipeline(trigger):
|
||||
name = "rrc-trigger-downstream",
|
||||
trigger = trigger,
|
||||
steps = steps,
|
||||
depends_on = ["rrc-integration-tests"],
|
||||
environment = environment,
|
||||
)
|
||||
|
||||
@@ -14,7 +14,6 @@ load(
|
||||
"compile_build_cmd",
|
||||
"download_grabpl_step",
|
||||
"e2e_tests_artifacts",
|
||||
"e2e_tests_step",
|
||||
"enterprise_downstream_step",
|
||||
"frontend_metrics_step",
|
||||
"grafana_server_step",
|
||||
@@ -38,10 +37,6 @@ load(
|
||||
"scripts/drone/steps/rgm.star",
|
||||
"rgm_artifacts_step",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/utils/images.star",
|
||||
"images",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/utils/utils.star",
|
||||
"pipeline",
|
||||
@@ -74,7 +69,6 @@ def build_e2e(trigger, ver_mode):
|
||||
build_steps = []
|
||||
|
||||
create_packages = rgm_artifacts_step(
|
||||
alpine = images["alpine"],
|
||||
artifacts = [
|
||||
"targz:grafana:linux/amd64",
|
||||
"targz:grafana:linux/arm64",
|
||||
@@ -88,7 +82,6 @@ def build_e2e(trigger, ver_mode):
|
||||
],
|
||||
file = "packages.txt",
|
||||
tag_format = "{{ .version_base }}-{{ .buildID }}-{{ .arch }}",
|
||||
ubuntu = images["ubuntu"],
|
||||
ubuntu_tag_format = "{{ .version_base }}-{{ .buildID }}-ubuntu-{{ .arch }}",
|
||||
)
|
||||
|
||||
@@ -122,14 +115,8 @@ def build_e2e(trigger, ver_mode):
|
||||
publish_docker,
|
||||
build_test_plugins_step(),
|
||||
grafana_server_step(),
|
||||
e2e_tests_step("dashboards-suite"),
|
||||
e2e_tests_step("old-arch/dashboards-suite"),
|
||||
e2e_tests_step("smoke-tests-suite"),
|
||||
e2e_tests_step("old-arch/smoke-tests-suite"),
|
||||
e2e_tests_step("panels-suite"),
|
||||
e2e_tests_step("old-arch/panels-suite"),
|
||||
e2e_tests_step("various-suite"),
|
||||
e2e_tests_step("old-arch/various-suite"),
|
||||
# Note: Main E2E test suites (dashboards, panels, smoke-tests, various) have been migrated to GitHub Actions
|
||||
# Only keeping tests that are not yet covered by GitHub Actions
|
||||
cloud_plugins_e2e_tests_step(
|
||||
"cloud-plugins-suite",
|
||||
cloud = "azure",
|
||||
@@ -138,7 +125,7 @@ def build_e2e(trigger, ver_mode):
|
||||
playwright_e2e_tests_step(),
|
||||
playwright_e2e_report_upload(),
|
||||
playwright_e2e_report_post_link(),
|
||||
e2e_tests_artifacts(),
|
||||
e2e_tests_artifacts(), # Collects artifacts from remaining E2E tests
|
||||
build_storybook_step(ver_mode = ver_mode),
|
||||
test_a11y_frontend_step(ver_mode = ver_mode),
|
||||
],
|
||||
|
||||
@@ -35,7 +35,6 @@ def enterprise_downstream_pipeline():
|
||||
]
|
||||
deps = [
|
||||
"main-build-e2e-publish",
|
||||
"main-integration-tests",
|
||||
]
|
||||
return pipeline(
|
||||
name = "main-trigger-downstream",
|
||||
|
||||
+16
-28
@@ -1,21 +1,15 @@
|
||||
"""
|
||||
rgm uses 'github.com/grafana/grafana-build' to build Grafana on the following events:
|
||||
* A merge to main
|
||||
* A tag that begins with a 'v'
|
||||
'rgm' pipelines are pipelines that use dagger (located in 'pkg/build/daggerbuild')
|
||||
"""
|
||||
|
||||
load(
|
||||
"scripts/drone/dagger.star",
|
||||
"with_dagger_install",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/events/release.star",
|
||||
"verify_release_pipeline",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_backend.star",
|
||||
"test_backend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/pipelines/test_frontend.star",
|
||||
"test_frontend",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/steps/github.star",
|
||||
"github_app_generate_token_step",
|
||||
@@ -33,7 +27,7 @@ load(
|
||||
)
|
||||
load(
|
||||
"scripts/drone/variables.star",
|
||||
"golang_version",
|
||||
"dagger_version",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/vault.star",
|
||||
@@ -132,19 +126,18 @@ def rgm_run(name, script):
|
||||
Drone step.
|
||||
"""
|
||||
env = {
|
||||
"GO_VERSION": golang_version,
|
||||
"ALPINE_BASE": images["alpine"],
|
||||
"UBUNTU_BASE": images["ubuntu"],
|
||||
}
|
||||
rgm_run_step = {
|
||||
"name": name,
|
||||
"image": "grafana/grafana-build:main",
|
||||
"image": images["go"],
|
||||
"pull": "always",
|
||||
"commands": [
|
||||
"commands": with_dagger_install([
|
||||
"export GRAFANA_DIR=$$(pwd)",
|
||||
"export GITHUB_TOKEN=$(cat /github-app/token)",
|
||||
"cd /src && ./scripts/{}".format(script),
|
||||
],
|
||||
"./pkg/build/daggerbuild/scripts/{}".format(script),
|
||||
], dagger_version),
|
||||
"environment": rgm_env_secrets(env),
|
||||
# The docker socket is a requirement for running dagger programs
|
||||
# In the future we should find a way to use dagger without mounting the docker socket.
|
||||
@@ -214,7 +207,6 @@ def rgm_main():
|
||||
name = "rgm-main-prerelease",
|
||||
trigger = main_trigger,
|
||||
steps = rgm_run("rgm-build", "drone_build_main.sh"),
|
||||
depends_on = ["main-test-backend"],
|
||||
)
|
||||
|
||||
def rgm_tag():
|
||||
@@ -251,7 +243,6 @@ def rgm_nightly_build():
|
||||
name = "rgm-nightly-build",
|
||||
trigger = nightly_trigger,
|
||||
steps = rgm_run("rgm-build", "drone_build_nightly_grafana.sh") + copy_steps,
|
||||
depends_on = ["nightly-test-backend", "nightly-test-frontend"],
|
||||
)
|
||||
|
||||
def rgm_nightly_publish():
|
||||
@@ -277,8 +268,6 @@ def rgm_nightly_publish():
|
||||
|
||||
def rgm_nightly_pipeline():
|
||||
return [
|
||||
test_frontend(nightly_trigger, "nightly"),
|
||||
test_backend(nightly_trigger, "nightly"),
|
||||
rgm_nightly_build(),
|
||||
rgm_nightly_publish(),
|
||||
]
|
||||
@@ -328,7 +317,6 @@ def rgm_promotion_pipeline():
|
||||
}
|
||||
|
||||
env = {
|
||||
"GO_VERSION": golang_version,
|
||||
"ALPINE_BASE": images["alpine"],
|
||||
"UBUNTU_BASE": images["ubuntu"],
|
||||
}
|
||||
@@ -342,18 +330,18 @@ def rgm_promotion_pipeline():
|
||||
# * UPLOAD_TO = Google Cloud Storage URL to upload the built artifacts to. (ex: gs://some-bucket/path)
|
||||
build_step = {
|
||||
"name": "rgm-build",
|
||||
"image": "grafana/grafana-build:main",
|
||||
"image": images["go"],
|
||||
"pull": "always",
|
||||
"commands": [
|
||||
"commands": with_dagger_install([
|
||||
"export GITHUB_TOKEN=$(cat /github-app/token)",
|
||||
"dagger run --silent /src/grafana-build artifacts " +
|
||||
"dagger run --silent go run ./pkg/build/cmd artifacts " +
|
||||
"-a $${ARTIFACTS} " +
|
||||
"--grafana-ref=$${GRAFANA_REF} " +
|
||||
"--enterprise-ref=$${ENTERPRISE_REF} " +
|
||||
"--grafana-repo=$${GRAFANA_REPO} " +
|
||||
"--version=$${VERSION} " +
|
||||
"--go-version={}".format(golang_version),
|
||||
],
|
||||
"--build-id=$${DRONE_BUILD_NUMBER} " +
|
||||
"--version=$${VERSION}",
|
||||
], dagger_version),
|
||||
"environment": rgm_env_secrets(env),
|
||||
# The docker socket is a requirement for running dagger programs
|
||||
# In the future we should find a way to use dagger without mounting the docker socket.
|
||||
|
||||
+26
-234
@@ -315,14 +315,16 @@ def store_storybook_step(ver_mode, trigger = None):
|
||||
return step
|
||||
|
||||
def e2e_tests_artifacts():
|
||||
# Note: This function is kept for backward compatibility but now only handles
|
||||
# artifacts from the remaining E2E tests that haven't been migrated to GitHub Actions
|
||||
return {
|
||||
"name": "e2e-tests-artifacts-upload",
|
||||
"image": images["cloudsdk"],
|
||||
"depends_on": [
|
||||
"end-to-end-tests-dashboards-suite",
|
||||
"end-to-end-tests-panels-suite",
|
||||
"end-to-end-tests-smoke-tests-suite",
|
||||
"end-to-end-tests-various-suite",
|
||||
# Note: Main E2E tests have been migrated to GitHub Actions
|
||||
# Only depend on remaining Drone E2E tests
|
||||
"end-to-end-tests-cloud-plugins-suite-azure",
|
||||
"playwright-plugin-e2e",
|
||||
github_app_generate_token_step()["name"],
|
||||
],
|
||||
"failure": "ignore",
|
||||
@@ -338,8 +340,8 @@ def e2e_tests_artifacts():
|
||||
},
|
||||
"commands": [
|
||||
"export GITHUB_TOKEN=$(cat /github-app/token)",
|
||||
# if no videos found do nothing
|
||||
"if [ -z `find ./e2e -type f -name *spec.ts.mp4` ]; then echo 'missing videos'; false; fi",
|
||||
# if no videos found do nothing (may be fewer videos now that main tests are in GitHub Actions)
|
||||
"if [ -z `find ./e2e -type f -name *spec.ts.mp4` ]; then echo 'no e2e videos found from remaining tests'; exit 0; fi",
|
||||
"apt-get update",
|
||||
"apt-get install -yq zip",
|
||||
"printenv GCP_GRAFANA_UPLOAD_ARTIFACTS_KEY > /tmp/gcpkey_upload_artifacts.json",
|
||||
@@ -568,34 +570,6 @@ def build_plugins_step(ver_mode):
|
||||
],
|
||||
}
|
||||
|
||||
def test_backend_step():
|
||||
return {
|
||||
"name": "test-backend",
|
||||
"image": images["go"],
|
||||
"depends_on": [
|
||||
"wire-install",
|
||||
],
|
||||
"commands": [
|
||||
# shared-mime-info and shared-mime-info-lang is used for exactly 1 test for the
|
||||
# mime.TypeByExtension function.
|
||||
"apk add --update build-base shared-mime-info shared-mime-info-lang",
|
||||
"go list -f '{{.Dir}}/...' -m | xargs go test -short -covermode=atomic -timeout=5m",
|
||||
],
|
||||
}
|
||||
|
||||
def test_backend_integration_step():
|
||||
return {
|
||||
"name": "test-backend-integration",
|
||||
"image": images["go"],
|
||||
"depends_on": [
|
||||
"wire-install",
|
||||
],
|
||||
"commands": [
|
||||
"apk add --update build-base",
|
||||
"go test -count=1 -covermode=atomic -timeout=5m -run '^TestIntegration' $(find ./pkg -type f -name '*_test.go' -exec grep -l '^func TestIntegration' '{}' '+' | grep -o '\\(.*\\)/' | sort -u)",
|
||||
],
|
||||
}
|
||||
|
||||
def betterer_frontend_step():
|
||||
"""Run betterer on frontend code.
|
||||
|
||||
@@ -615,44 +589,6 @@ def betterer_frontend_step():
|
||||
],
|
||||
}
|
||||
|
||||
def test_frontend_step():
|
||||
"""Runs tests on frontend code.
|
||||
|
||||
Returns:
|
||||
Drone step.
|
||||
"""
|
||||
|
||||
return {
|
||||
"name": "test-frontend",
|
||||
"image": images["node"],
|
||||
"environment": {
|
||||
"TEST_MAX_WORKERS": "50%",
|
||||
},
|
||||
"depends_on": [
|
||||
"yarn-install",
|
||||
],
|
||||
"commands": [
|
||||
"yarn run ci:test-frontend",
|
||||
],
|
||||
}
|
||||
|
||||
def lint_frontend_step():
|
||||
return {
|
||||
"name": "lint-frontend",
|
||||
"image": images["node"],
|
||||
"environment": {
|
||||
"TEST_MAX_WORKERS": "50%",
|
||||
},
|
||||
"depends_on": [
|
||||
"yarn-install",
|
||||
],
|
||||
"commands": [
|
||||
"yarn run prettier:check",
|
||||
"yarn run lint",
|
||||
"yarn run typecheck",
|
||||
],
|
||||
}
|
||||
|
||||
def verify_i18n_step():
|
||||
extract_error_message = "\nExtraction failed. Make sure that you have no dynamic translation phrases, such as 't(\\`preferences.theme.\\$${themeID}\\`, themeName)' and that no translation key is used twice. Search the output for '[warning]' to find the offending file."
|
||||
uncommited_error_message = "\nTranslation extraction has not been committed. Please run 'make i18n-extract', commit the changes and push again."
|
||||
@@ -713,21 +649,13 @@ def test_a11y_frontend_step(ver_mode, port = 3001):
|
||||
commands = [
|
||||
# Note - this runs in a container running node 14, which does not support the -y option to npx
|
||||
"npx wait-on@7.0.1 http://$HOST:$PORT",
|
||||
"pa11y-ci --config e2e/pa11yci.conf.js",
|
||||
]
|
||||
failure = "ignore"
|
||||
no_thresholds = "true"
|
||||
if ver_mode == "pr":
|
||||
commands.extend(
|
||||
[
|
||||
"pa11y-ci --config .pa11yci-pr.conf.js",
|
||||
],
|
||||
)
|
||||
failure = "always"
|
||||
else:
|
||||
commands.extend(
|
||||
[
|
||||
"pa11y-ci --config .pa11yci.conf.js --json > pa11y-ci-results.json",
|
||||
],
|
||||
)
|
||||
no_thresholds = "false"
|
||||
|
||||
return {
|
||||
"name": "test-a11y-frontend",
|
||||
@@ -740,6 +668,7 @@ def test_a11y_frontend_step(ver_mode, port = 3001):
|
||||
"GRAFANA_MISC_STATS_API_KEY": from_secret("grafana_misc_stats_api_key"),
|
||||
"HOST": "grafana-server",
|
||||
"PORT": port,
|
||||
"NO_THRESHOLDS": no_thresholds,
|
||||
},
|
||||
"failure": failure,
|
||||
"commands": commands,
|
||||
@@ -835,23 +764,6 @@ def start_storybook_step():
|
||||
"detach": True,
|
||||
}
|
||||
|
||||
def e2e_storybook_step():
|
||||
return {
|
||||
"name": "end-to-end-tests-storybook-suite",
|
||||
"image": images["cypress"],
|
||||
"depends_on": [
|
||||
"start-storybook",
|
||||
],
|
||||
"environment": {
|
||||
"HOST": "start-storybook",
|
||||
"PORT": "9001",
|
||||
},
|
||||
"commands": [
|
||||
"npx wait-on@7.2.0 -t 1m http://$HOST:$PORT",
|
||||
"yarn e2e:storybook",
|
||||
],
|
||||
}
|
||||
|
||||
def cloud_plugins_e2e_tests_step(suite, cloud, trigger = None):
|
||||
"""Run cloud plugins end-to-end tests.
|
||||
|
||||
@@ -890,7 +802,7 @@ def cloud_plugins_e2e_tests_step(suite, cloud, trigger = None):
|
||||
branch = "${DRONE_SOURCE_BRANCH}".replace("/", "-")
|
||||
step = {
|
||||
"name": "end-to-end-tests-{}-{}".format(suite, cloud),
|
||||
"image": "us-docker.pkg.dev/grafanalabs-dev/cloud-data-sources/e2e-13.10.0:1.0.0",
|
||||
"image": "us-docker.pkg.dev/grafanalabs-dev/docker-oss-plugin-partnerships-dev/e2e-14.3.2:1.0.0",
|
||||
"depends_on": [
|
||||
"grafana-server",
|
||||
github_app_generate_token_step()["name"],
|
||||
@@ -1014,129 +926,6 @@ def publish_images_step(ver_mode, docker_repo, trigger = None, depends_on = ["rg
|
||||
|
||||
return step
|
||||
|
||||
def integration_tests_steps(name, cmds, hostname = None, port = None, environment = None, canFail = False):
|
||||
"""Integration test steps
|
||||
|
||||
Args:
|
||||
name: the name of the step.
|
||||
cmds: the commands to run to perform the integration tests.
|
||||
hostname: the hostname where the remote server is available.
|
||||
port: the port where the remote server is available.
|
||||
environment: Any extra environment variables needed to run the integration tests.
|
||||
canFail: controls whether the step can fail.
|
||||
|
||||
Returns:
|
||||
A list of drone steps. If a hostname / port were provided, then a step to wait for the remove server to be
|
||||
available is also returned.
|
||||
"""
|
||||
dockerize_name = "wait-for-{}".format(name)
|
||||
|
||||
depends = [
|
||||
"wire-install",
|
||||
]
|
||||
|
||||
step = {
|
||||
"name": "{}-integration-tests".format(name),
|
||||
"image": images["go"],
|
||||
"depends_on": depends,
|
||||
"commands": [
|
||||
"apk add --update build-base",
|
||||
] + cmds,
|
||||
}
|
||||
|
||||
if canFail:
|
||||
step["failure"] = "ignore"
|
||||
|
||||
if environment:
|
||||
step["environment"] = environment
|
||||
|
||||
if hostname == None:
|
||||
return [step]
|
||||
|
||||
depends = depends.append(dockerize_name)
|
||||
|
||||
return [
|
||||
dockerize_step(dockerize_name, hostname, port),
|
||||
step,
|
||||
]
|
||||
|
||||
def integration_benchmarks_step(name, environment = None):
|
||||
cmds = [
|
||||
"if [ -z ${GO_PACKAGES} ]; then echo 'missing GO_PACKAGES'; false; fi",
|
||||
"go test -v -run=^$ -benchmem -timeout=1h -count=8 -bench=. ${GO_PACKAGES}",
|
||||
]
|
||||
|
||||
return integration_tests_steps("{}-benchmark".format(name), cmds, environment = environment)
|
||||
|
||||
def postgres_integration_tests_steps():
|
||||
cmds = [
|
||||
"apk add --update postgresql-client",
|
||||
"psql -p 5432 -h postgres -U grafanatest -d grafanatest -f " +
|
||||
"devenv/docker/blocks/postgres_tests/setup.sql",
|
||||
"go clean -testcache",
|
||||
"go test -p=1 -count=1 -covermode=atomic -timeout=5m -run '^TestIntegration' $(find ./pkg -type f -name '*_test.go' -exec grep -l '^func TestIntegration' '{}' '+' | grep -o '\\(.*\\)/' | sort -u)",
|
||||
]
|
||||
|
||||
environment = {
|
||||
"PGPASSWORD": "grafanatest",
|
||||
"GRAFANA_TEST_DB": "postgres",
|
||||
"POSTGRES_HOST": "postgres",
|
||||
}
|
||||
|
||||
return integration_tests_steps("postgres", cmds, "postgres", "5432", environment)
|
||||
|
||||
def mysql_integration_tests_steps(hostname, version):
|
||||
cmds = [
|
||||
"apk add --update mariadb-client", # alpine doesn't package mysql anymore; more info: https://wiki.alpinelinux.org/wiki/MySQL
|
||||
"cat devenv/docker/blocks/mysql_tests/setup.sql | mariadb -h {} -P 3306 -u root -prootpass --disable-ssl-verify-server-cert".format(hostname),
|
||||
"go clean -testcache",
|
||||
"go test -p=1 -count=1 -covermode=atomic -timeout=5m -run '^TestIntegration' $(find ./pkg -type f -name '*_test.go' -exec grep -l '^func TestIntegration' '{}' '+' | grep -o '\\(.*\\)/' | sort -u)",
|
||||
]
|
||||
|
||||
environment = {
|
||||
"GRAFANA_TEST_DB": "mysql",
|
||||
"MYSQL_HOST": hostname,
|
||||
}
|
||||
|
||||
return integration_tests_steps("mysql-{}".format(version), cmds, hostname, "3306", environment)
|
||||
|
||||
def redis_integration_tests_steps():
|
||||
cmds = [
|
||||
"go clean -testcache",
|
||||
"go list -f '{{.Dir}}/...' -m | xargs go test -run IntegrationRedis -covermode=atomic -timeout=2m",
|
||||
]
|
||||
|
||||
environment = {
|
||||
"REDIS_URL": "redis://redis:6379/0",
|
||||
}
|
||||
|
||||
return integration_tests_steps("redis", cmds, "redis", "6379", environment = environment)
|
||||
|
||||
def remote_alertmanager_integration_tests_steps():
|
||||
cmds = [
|
||||
"go clean -testcache",
|
||||
"go test -run TestIntegrationRemoteAlertmanager -covermode=atomic -timeout=2m ./pkg/services/ngalert/...",
|
||||
]
|
||||
|
||||
environment = {
|
||||
"AM_TENANT_ID": "test",
|
||||
"AM_URL": "http://mimir_backend:8080",
|
||||
}
|
||||
|
||||
return integration_tests_steps("remote-alertmanager", cmds, "mimir_backend", "8080", environment = environment)
|
||||
|
||||
def memcached_integration_tests_steps():
|
||||
cmds = [
|
||||
"go clean -testcache",
|
||||
"go list -f '{{.Dir}}/...' -m | xargs go test -run IntegrationMemcached -covermode=atomic -timeout=2m",
|
||||
]
|
||||
|
||||
environment = {
|
||||
"MEMCACHED_HOSTS": "memcached:11211",
|
||||
}
|
||||
|
||||
return integration_tests_steps("memcached", cmds, "memcached", "11211", environment)
|
||||
|
||||
def release_canary_npm_packages_step(trigger = None):
|
||||
"""Releases canary NPM packages.
|
||||
|
||||
@@ -1175,12 +964,15 @@ def release_canary_npm_packages_step(trigger = None):
|
||||
|
||||
return step
|
||||
|
||||
def upload_packages_step(ver_mode, trigger = None, depends_on = [
|
||||
"end-to-end-tests-dashboards-suite",
|
||||
"end-to-end-tests-panels-suite",
|
||||
"end-to-end-tests-smoke-tests-suite",
|
||||
"end-to-end-tests-various-suite",
|
||||
]):
|
||||
def upload_packages_step(
|
||||
ver_mode,
|
||||
trigger = None,
|
||||
depends_on = [
|
||||
# Note: Main E2E tests have been migrated to GitHub Actions
|
||||
# Updated dependencies to only include remaining Drone E2E tests
|
||||
"end-to-end-tests-cloud-plugins-suite-azure",
|
||||
"playwright-plugin-e2e",
|
||||
]):
|
||||
"""Upload packages to object storage.
|
||||
|
||||
Args:
|
||||
@@ -1341,11 +1133,11 @@ def verify_gen_jsonnet_step():
|
||||
}
|
||||
|
||||
def end_to_end_tests_deps():
|
||||
# Note: Main E2E tests have been migrated to GitHub Actions
|
||||
# Only return dependencies for E2E tests that still run in Drone
|
||||
return [
|
||||
"end-to-end-tests-dashboards-suite",
|
||||
"end-to-end-tests-panels-suite",
|
||||
"end-to-end-tests-smoke-tests-suite",
|
||||
"end-to-end-tests-various-suite",
|
||||
"end-to-end-tests-cloud-plugins-suite-azure",
|
||||
"playwright-plugin-e2e",
|
||||
]
|
||||
|
||||
def compile_build_cmd():
|
||||
|
||||
@@ -3,13 +3,17 @@ Individual steps that use 'grafana-build' to replace existing individual steps.
|
||||
These aren't used in releases.
|
||||
"""
|
||||
|
||||
load(
|
||||
"scripts/drone/dagger.star",
|
||||
"with_dagger_install",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/utils/images.star",
|
||||
"images",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/variables.star",
|
||||
"golang_version",
|
||||
"dagger_version",
|
||||
)
|
||||
load(
|
||||
"scripts/drone/vault.star",
|
||||
@@ -18,7 +22,7 @@ load(
|
||||
)
|
||||
|
||||
def artifacts_cmd(artifacts = []):
|
||||
cmd = "/src/grafana-build artifacts "
|
||||
cmd = "dagger run go run ./pkg/build/cmd artifacts "
|
||||
|
||||
for artifact in artifacts:
|
||||
cmd += "-a {} ".format(artifact)
|
||||
@@ -33,25 +37,24 @@ def rgm_artifacts_step(
|
||||
depends_on = ["yarn-install"],
|
||||
tag_format = "{{ .version }}-{{ .arch }}",
|
||||
ubuntu_tag_format = "{{ .version }}-ubuntu-{{ .arch }}",
|
||||
verify = "false",
|
||||
ubuntu = images["ubuntu"],
|
||||
alpine = images["alpine"]):
|
||||
alpine = images["alpine"],
|
||||
verify = "false"):
|
||||
cmd = artifacts_cmd(artifacts = artifacts)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"image": "grafana/grafana-build:main",
|
||||
"image": images["go"],
|
||||
"pull": "always",
|
||||
"depends_on": depends_on,
|
||||
"environment": {
|
||||
"_EXPERIMENTAL_DAGGER_CLOUD_TOKEN": from_secret(rgm_dagger_token),
|
||||
},
|
||||
"commands": [
|
||||
"commands": with_dagger_install([
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --version",
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --uninstall 'qemu-*'",
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --install all",
|
||||
cmd +
|
||||
"--go-version={} ".format(golang_version) +
|
||||
"--yarn-cache=$$YARN_CACHE_FOLDER " +
|
||||
"--build-id=$$DRONE_BUILD_NUMBER " +
|
||||
"--ubuntu-base={} ".format(ubuntu) +
|
||||
@@ -61,29 +64,27 @@ def rgm_artifacts_step(
|
||||
"--verify='{}' ".format(verify) +
|
||||
"--grafana-dir=$$PWD > {}".format(file),
|
||||
"find ./dist -name '*docker*.tar.gz' -type f | xargs -n1 docker load -i",
|
||||
],
|
||||
], dagger_version),
|
||||
"volumes": [{"name": "docker", "path": "/var/run/docker.sock"}],
|
||||
}
|
||||
|
||||
# rgm_build_backend will create compile the grafana backend for various platforms. It's preferred to use
|
||||
# 'rgm_package_step' if you creating a "usable" artifact. This should really only be used to verify that the code is
|
||||
# compilable.
|
||||
# rgm_build_backend will create compile the grafana backend for various platforms.
|
||||
def rgm_build_backend_step(artifacts = ["backend:grafana:linux/amd64", "backend:grafana:linux/arm64"]):
|
||||
return rgm_artifacts_step(name = "rgm-build-backend", artifacts = artifacts, depends_on = [])
|
||||
|
||||
def rgm_build_docker_step(ubuntu, alpine, depends_on = ["yarn-install"], file = "docker.txt", tag_format = "{{ .version }}-{{ .arch }}", ubuntu_tag_format = "{{ .version }}-ubuntu-{{ .arch }}"):
|
||||
def rgm_build_docker_step(depends_on = ["yarn-install"], file = "docker.txt", tag_format = "{{ .version }}-{{ .arch }}", ubuntu_tag_format = "{{ .version }}-ubuntu-{{ .arch }}", ubuntu = images["ubuntu"], alpine = images["alpine"]):
|
||||
return {
|
||||
"name": "rgm-build-docker",
|
||||
"image": "grafana/grafana-build:main",
|
||||
"image": images["go"],
|
||||
"pull": "always",
|
||||
"environment": {
|
||||
"_EXPERIMENTAL_DAGGER_CLOUD_TOKEN": from_secret(rgm_dagger_token),
|
||||
},
|
||||
"commands": [
|
||||
"commands": with_dagger_install([
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --version",
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --uninstall 'qemu-*'",
|
||||
"docker run --privileged --rm tonistiigi/binfmt:qemu-v7.0.0-28 --install all",
|
||||
"/src/grafana-build artifacts " +
|
||||
"dagger run go run ./pkg/build/cmd artifacts " +
|
||||
"-a docker:grafana:linux/amd64 " +
|
||||
"-a docker:grafana:linux/amd64:ubuntu " +
|
||||
"-a docker:grafana:linux/arm64 " +
|
||||
@@ -92,14 +93,13 @@ def rgm_build_docker_step(ubuntu, alpine, depends_on = ["yarn-install"], file =
|
||||
"-a docker:grafana:linux/arm/v7:ubuntu " +
|
||||
"--yarn-cache=$$YARN_CACHE_FOLDER " +
|
||||
"--build-id=$$DRONE_BUILD_NUMBER " +
|
||||
"--go-version={} ".format(golang_version) +
|
||||
"--ubuntu-base={} ".format(ubuntu) +
|
||||
"--alpine-base={} ".format(alpine) +
|
||||
"--tag-format='{}' ".format(tag_format) +
|
||||
"--grafana-dir=$$PWD " +
|
||||
"--ubuntu-tag-format='{}' > {}".format(ubuntu_tag_format, file),
|
||||
"find ./dist -name '*docker*.tar.gz' -type f | xargs -n1 docker load -i",
|
||||
],
|
||||
], dagger_version),
|
||||
"volumes": [{"name": "docker", "path": "/var/run/docker.sock"}],
|
||||
"depends_on": depends_on,
|
||||
}
|
||||
|
||||
@@ -20,21 +20,11 @@ images = {
|
||||
"ubuntu": "ubuntu:22.04",
|
||||
"curl": "byrnedo/alpine-curl:0.1.8",
|
||||
"plugins_slack": "plugins/slack",
|
||||
"python": "python:3.8",
|
||||
"postgres_alpine": "postgres:12.3-alpine",
|
||||
"mimir": "grafana/mimir-alpine:r316-55f47f8",
|
||||
"mysql8": "mysql:8.0.32",
|
||||
"redis_alpine": "redis:6.2.11-alpine",
|
||||
"memcached_alpine": "memcached:1.6.9-alpine",
|
||||
"package_publish": "us.gcr.io/kubernetes-dev/package-publish:latest",
|
||||
"openldap": "osixia/openldap:1.4.0",
|
||||
"drone_downstream": "grafana/drone-downstream",
|
||||
"docker_puppeteer": "grafana/docker-puppeteer:1.1.0",
|
||||
"docs": "grafana/docs-base:latest",
|
||||
"cypress": "cypress/included:13.10.0",
|
||||
"cypress": "cypress/included:14.3.2",
|
||||
"dockerize": "jwilder/dockerize:0.6.1",
|
||||
"shellcheck": "koalaman/shellcheck:stable",
|
||||
"rocky": "rockylinux:9",
|
||||
"wine": "scottyhardy/docker-wine:stable-9.0",
|
||||
"github_app_secret_writer": "us-docker.pkg.dev/grafanalabs-global/docker-deployment-tools-prod/github-app-secret-writer:2024-11-05-v11688112090.1-83920c59",
|
||||
}
|
||||
|
||||
@@ -6,4 +6,5 @@ grabpl_version = "v3.1.2"
|
||||
golang_version = "1.24.4"
|
||||
|
||||
# nodejs_version should match what's in ".nvmrc", but without the v prefix.
|
||||
nodejs_version = "22.11.0"
|
||||
nodejs_version = "22.16.0"
|
||||
dagger_version = "v0.18.8"
|
||||
|
||||
@@ -41,31 +41,45 @@ const config: ConfigFile = {
|
||||
apiImport: 'baseAPI',
|
||||
filterEndpoints: ['getUserPreferences', 'updateUserPreferences', 'patchUserPreferences'],
|
||||
},
|
||||
'../public/app/api/clients/iam/endpoints.gen.ts': {
|
||||
'../public/app/api/clients/iam/v0alpha1/endpoints.gen.ts': {
|
||||
schemaFile: '../data/openapi/iam.grafana.app-v0alpha1.json',
|
||||
apiFile: '../public/app/api/clients/iam/baseAPI.ts',
|
||||
apiFile: '../public/app/api/clients/iam/v0alpha1/baseAPI.ts',
|
||||
filterEndpoints: ['getDisplayMapping'],
|
||||
tag: true,
|
||||
},
|
||||
'../public/app/api/clients/provisioning/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/provisioning/baseAPI.ts',
|
||||
'../public/app/api/clients/provisioning/v0alpha1/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/provisioning/v0alpha1/baseAPI.ts',
|
||||
schemaFile: '../data/openapi/provisioning.grafana.app-v0alpha1.json',
|
||||
filterEndpoints,
|
||||
tag: true,
|
||||
hooks: true,
|
||||
},
|
||||
'../public/app/api/clients/folder/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/folder/baseAPI.ts',
|
||||
'../public/app/api/clients/folder/v1beta1/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/folder/v1beta1/baseAPI.ts',
|
||||
schemaFile: '../data/openapi/folder.grafana.app-v1beta1.json',
|
||||
filterEndpoints: ['getFolder'],
|
||||
tag: true,
|
||||
},
|
||||
'../public/app/api/clients/advisor/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/advisor/baseAPI.ts',
|
||||
'../public/app/api/clients/advisor/v0alpha1/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/advisor/v0alpha1/baseAPI.ts',
|
||||
schemaFile: '../data/openapi/advisor.grafana.app-v0alpha1.json',
|
||||
filterEndpoints: ['createCheck', 'getCheck', 'listCheck', 'deleteCheck', 'updateCheck', 'listCheckType'],
|
||||
filterEndpoints: [
|
||||
'createCheck',
|
||||
'getCheck',
|
||||
'listCheck',
|
||||
'deleteCheck',
|
||||
'updateCheck',
|
||||
'listCheckType',
|
||||
'updateCheckType',
|
||||
],
|
||||
tag: true,
|
||||
},
|
||||
'../public/app/api/clients/playlist/v0alpha1/endpoints.gen.ts': {
|
||||
apiFile: '../public/app/api/clients/playlist/v0alpha1/baseAPI.ts',
|
||||
schemaFile: '../data/openapi/playlist.grafana.app-v0alpha1.json',
|
||||
filterEndpoints: ['listPlaylist', 'getPlaylist', 'createPlaylist', 'deletePlaylist', 'replacePlaylist'],
|
||||
tag: true,
|
||||
},
|
||||
// PLOP_INJECT_API_CLIENT - Used by the API client generator
|
||||
},
|
||||
};
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
|
||||
. scripts/grafana-server/variables
|
||||
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
set -eo pipefail
|
||||
|
||||
. scripts/grafana-server/variables
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
|
||||
DEFAULT_RUNDIR=scripts/grafana-server/tmp
|
||||
RUNDIR=${RUNDIR:-$DEFAULT_RUNDIR}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
#!/bin/bash
|
||||
#!/usr/bin/env bash
|
||||
set -eo pipefail
|
||||
|
||||
. scripts/grafana-server/variables
|
||||
|
||||
@@ -4,6 +4,8 @@ const printAffectedPluginsSection = require('./levitate-show-affected-plugins');
|
||||
|
||||
const data = JSON.parse(fs.readFileSync('data.json', 'utf8'));
|
||||
|
||||
const isFork = Boolean(process.env.IS_FORK || false);
|
||||
|
||||
function stripAnsi(str) {
|
||||
return str.replace(/\x1b\[[0-9;]*m/g, '');
|
||||
}
|
||||
@@ -30,7 +32,8 @@ if (data.changes.length > 0) {
|
||||
markdown += printSection('Changes', data.changes);
|
||||
}
|
||||
|
||||
if (data.removals.length > 0 || data.changes.length > 0) {
|
||||
// The logic below would need access to secrets for accessing BigQuery, however that's not available on forks.
|
||||
if ((data.removals.length > 0 || data.changes.length > 0) && !isFork) {
|
||||
markdown += printAffectedPluginsSection(data);
|
||||
}
|
||||
|
||||
|
||||
@@ -1,6 +1,5 @@
|
||||
import PackageJson from '@npmcli/package-json';
|
||||
import { mkdir } from 'node:fs/promises';
|
||||
import { join, dirname } from 'node:path';
|
||||
|
||||
const cwd = process.cwd();
|
||||
|
||||
@@ -8,18 +7,17 @@ try {
|
||||
const pkgJson = await PackageJson.load(cwd);
|
||||
const cjsIndex = pkgJson.content.publishConfig?.main ?? pkgJson.content.main;
|
||||
const esmIndex = pkgJson.content.publishConfig?.module ?? pkgJson.content.module;
|
||||
const cjsTypes = pkgJson.content.publishConfig?.types ?? pkgJson.content.types;
|
||||
const esmTypes = `./${join(dirname(esmIndex), 'index.d.mts')}`;
|
||||
const typesIndex = pkgJson.content.publishConfig?.types ?? pkgJson.content.types;
|
||||
|
||||
const exports = {
|
||||
'./package.json': './package.json',
|
||||
'.': {
|
||||
import: {
|
||||
types: esmTypes,
|
||||
types: typesIndex,
|
||||
default: esmIndex,
|
||||
},
|
||||
require: {
|
||||
types: cjsTypes,
|
||||
types: typesIndex,
|
||||
default: cjsIndex,
|
||||
},
|
||||
},
|
||||
@@ -33,9 +31,17 @@ try {
|
||||
};
|
||||
}
|
||||
|
||||
// Fix for @grafana/i18n so eslint-plugin can be imported by consumers
|
||||
if (pkgJson.content.name === '@grafana/i18n') {
|
||||
exports['./eslint-plugin'] = {
|
||||
import: './dist/eslint/index.cjs',
|
||||
require: './dist/eslint/index.cjs',
|
||||
};
|
||||
}
|
||||
|
||||
pkgJson.update({
|
||||
main: cjsIndex,
|
||||
types: cjsTypes,
|
||||
types: typesIndex,
|
||||
module: esmIndex,
|
||||
exports,
|
||||
});
|
||||
@@ -52,11 +58,11 @@ try {
|
||||
...pkgJson.content.exports,
|
||||
[`./${aliasName}`]: {
|
||||
import: {
|
||||
types: esmTypes.replace('index', aliasName),
|
||||
types: typesIndex.replace('index', aliasName),
|
||||
default: esmIndex.replace('index', aliasName),
|
||||
},
|
||||
require: {
|
||||
types: cjsTypes.replace('index', aliasName),
|
||||
types: typesIndex.replace('index', aliasName),
|
||||
default: cjsIndex.replace('index', aliasName),
|
||||
},
|
||||
},
|
||||
@@ -80,7 +86,7 @@ async function createAliasPackageJsonFiles(packageJsonContent, aliasName) {
|
||||
const pkgJson = await PackageJson.create(pkgJsonPath, {
|
||||
data: {
|
||||
name: pkgName,
|
||||
types: `../dist/cjs/${aliasName}.d.cts`,
|
||||
types: `../dist/types/${aliasName}.d.ts`,
|
||||
main: `../dist/cjs/${aliasName}.cjs`,
|
||||
module: `../dist/esm/${aliasName}.mjs`,
|
||||
},
|
||||
|
||||
@@ -47,14 +47,14 @@ done
|
||||
|
||||
# Check if any files in packages/grafana-e2e-selectors were changed. If so, add a 'modified' tag to the package
|
||||
CHANGES_COUNT=$(git diff HEAD~1..HEAD --name-only -- packages/grafana-e2e-selectors | awk 'END{print NR}')
|
||||
if (( $CHANGES_COUNT > 0 )); then
|
||||
if (( CHANGES_COUNT > 0 )); then
|
||||
# Wait a little bit to allow the package to be published to the registry
|
||||
sleep 5s
|
||||
regex_pattern="canary: ([0-9.-]+)"
|
||||
TAGS=$(npm dist-tag ls @grafana/e2e-selectors)
|
||||
if [[ $TAGS =~ $regex_pattern ]]; then
|
||||
echo "$CHANGES_COUNT file(s) in packages/grafana-e2e-selectors were changed. Adding 'modified' tag to @grafana/e2e-selectors@${BASH_REMATCH[1]}"
|
||||
npm dist-tag add @grafana/e2e-selectors@${BASH_REMATCH[1]} modified
|
||||
npm dist-tag add @grafana/e2e-selectors@"${BASH_REMATCH[1]}" modified
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
@@ -61,7 +61,7 @@ echo " Date: $(git log -1 --format="%ad" --date=iso "$COMMIT_HASH")"
|
||||
|
||||
# Extract original PR number and create link
|
||||
PR_NUMBER=$(git log -1 --pretty=format:"%B" "$COMMIT_HASH" | grep -o '#[0-9]\+' | head -n1 | tr -d '#')
|
||||
if [ ! -z "$PR_NUMBER" ]; then
|
||||
if [ -n "$PR_NUMBER" ]; then
|
||||
# Extract PR title (first line of commit message)
|
||||
PR_TITLE=$(git log -1 --pretty=format:"%s" "$COMMIT_HASH")
|
||||
echo " PR: #$PR_NUMBER - $PR_TITLE"
|
||||
@@ -104,7 +104,7 @@ done
|
||||
if [ ${#direct_tags[@]} -gt 0 ] || [ ${#included_tags[@]} -gt 0 ]; then
|
||||
echo "This commit has been included in these PREVIOUS on-prem releases:"
|
||||
# Get all tags sorted
|
||||
all_tags=($(printf "%s\n" "${direct_tags[@]}" "${included_tags[@]}" | sort -V))
|
||||
readarray -t all_tags < <(printf "%s\n" "${direct_tags[@]}" "${included_tags[@]}" | sort -V)
|
||||
# Get the first release
|
||||
first_release="${all_tags[0]}"
|
||||
# Print all tags with annotation for the first release
|
||||
|
||||
@@ -0,0 +1,52 @@
|
||||
# RTK Query API Client Generator
|
||||
|
||||
This generator automates the process of creating RTK Query API clients for Grafana's API groups. It replaces the manual steps outlined in the [main API documentation](../../public/app/api/README.md).
|
||||
|
||||
## Usage
|
||||
|
||||
```bash
|
||||
yarn generate:api-client
|
||||
```
|
||||
|
||||
The CLI will prompt for:
|
||||
|
||||
1. **Enterprise or OSS API** - Whether this is an Enterprise or OSS API. This affects paths and build commands.
|
||||
2. **API group name** - The basic name for the API (e.g., `dashboard`)
|
||||
3. **API group** - The full API group name (defaults to `<group-name>.grafana.app`)
|
||||
4. **API version** - The API version (e.g., `v0alpha1`)
|
||||
5. **Reducer path** - The Redux reducer path (defaults to `<group-name>API`). This will also be used as the API's named export.
|
||||
6. **Endpoints** - Optional comma-separated list of endpoints to include (e.g., `createDashboard,updateDashboard`). If not provided, all endpoints will be included.
|
||||
|
||||
## What It Does
|
||||
|
||||
The generator automates the following:
|
||||
|
||||
1. Creates the `baseAPI.ts` file for the API group
|
||||
2. Updates the appropriate generate script to include the API client
|
||||
- `scripts/generate-rtk-apis.ts` for OSS APIs
|
||||
- `local/generate-enterprise-apis.ts` for Enterprise APIs
|
||||
3. Creates the `index.ts` file with proper exports
|
||||
4. For OSS APIs only: Registers Redux reducers and middleware in the store. For Enterprise this needs to be done manually
|
||||
5. Formats all generated files using Prettier and ESLint
|
||||
6. Automatically runs the appropriate command to generate endpoints from the OpenAPI schema
|
||||
|
||||
## Limitations
|
||||
|
||||
- The generator is optimized for Kubernetes-style APIs, as it requires Kubernetes resource details. For legacy APIs, manual adjustments may be needed.
|
||||
- It expects processed OpenAPI specifications to exist in the `openapi_snapshots` directory
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Missing OpenAPI Schema
|
||||
|
||||
If an error about a missing OpenAPI schema appears, check that:
|
||||
|
||||
1. The API group and version exist in the backend
|
||||
2. The `TestIntegrationOpenAPIs` test has been run to generate the schema (step 1 in the [main API documentation](../../public/app/api/README.md)).
|
||||
3. The schema file exists at `data/openapi/<group>-<version>.json`
|
||||
|
||||
### Validation Errors
|
||||
|
||||
- API group must include `.grafana.app`
|
||||
- Version must be in format `v0alpha1`, `v1beta2`, etc.
|
||||
- Reducer path must end with `API`
|
||||
@@ -0,0 +1,115 @@
|
||||
import { execSync } from 'child_process';
|
||||
import path from 'path';
|
||||
|
||||
type PlopActionFunction = (
|
||||
answers: Record<string, unknown>,
|
||||
config?: Record<string, unknown>
|
||||
) => string | Promise<string>;
|
||||
|
||||
// Helper to remove quotes from operation IDs
|
||||
export const removeQuotes = (str: string | unknown) => {
|
||||
if (typeof str !== 'string') {
|
||||
return str;
|
||||
}
|
||||
return str.replace(/^['"](.*)['"]$/, '$1');
|
||||
};
|
||||
|
||||
export const formatEndpoints = () => (endpointsInput: string | string[]) => {
|
||||
if (Array.isArray(endpointsInput)) {
|
||||
return endpointsInput.map((op) => `'${removeQuotes(op)}'`).join(', ');
|
||||
}
|
||||
|
||||
// Handle string input (comma-separated)
|
||||
if (typeof endpointsInput === 'string') {
|
||||
const endpointsArray = endpointsInput
|
||||
.split(',')
|
||||
.map((id) => id.trim())
|
||||
.filter(Boolean);
|
||||
|
||||
return endpointsArray.map((op) => `'${removeQuotes(op)}'`).join(', ');
|
||||
}
|
||||
|
||||
return '';
|
||||
};
|
||||
|
||||
// List of created or modified files
|
||||
export const getFilesToFormat = (groupName: string, version: string, isEnterprise = false) => {
|
||||
const apiClientBasePath = isEnterprise ? 'public/app/extensions/api/clients' : 'public/app/api/clients';
|
||||
const generateScriptPath = isEnterprise ? 'local/generate-enterprise-apis.ts' : 'scripts/generate-rtk-apis.ts';
|
||||
|
||||
return [
|
||||
`${apiClientBasePath}/${groupName}/${version}/baseAPI.ts`,
|
||||
`${apiClientBasePath}/${groupName}/${version}/index.ts`,
|
||||
generateScriptPath,
|
||||
...(isEnterprise ? [] : [`public/app/core/reducers/root.ts`, `public/app/store/configureStore.ts`]),
|
||||
];
|
||||
};
|
||||
|
||||
export const runGenerateApis =
|
||||
(basePath: string): PlopActionFunction =>
|
||||
(answers, config) => {
|
||||
try {
|
||||
const isEnterprise = answers.isEnterprise || (config && config.isEnterprise);
|
||||
|
||||
let command;
|
||||
if (isEnterprise) {
|
||||
command = 'yarn process-specs && npx rtk-query-codegen-openapi ./local/generate-enterprise-apis.ts';
|
||||
} else {
|
||||
command = 'yarn generate-apis';
|
||||
}
|
||||
|
||||
console.log(`⏳ Running ${command} to generate endpoints...`);
|
||||
execSync(command, { stdio: 'inherit', cwd: basePath });
|
||||
return '✅ API endpoints generated successfully!';
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
console.error('❌ Failed to generate API endpoints:', errorMessage);
|
||||
return '❌ Failed to generate API endpoints. See error above.';
|
||||
}
|
||||
};
|
||||
|
||||
export const formatFiles =
|
||||
(basePath: string): PlopActionFunction =>
|
||||
(_, config) => {
|
||||
if (!config || !Array.isArray(config.files)) {
|
||||
console.error('Invalid config passed to formatFiles action');
|
||||
return '❌ Formatting failed: Invalid configuration';
|
||||
}
|
||||
|
||||
const filesToFormat = config.files.map((file: string) => path.join(basePath, file));
|
||||
|
||||
try {
|
||||
const filesList = filesToFormat.map((file: string) => `"${file}"`).join(' ');
|
||||
|
||||
console.log('🧹 Running ESLint on generated/modified files...');
|
||||
try {
|
||||
execSync(`yarn eslint --fix ${filesList}`, { cwd: basePath });
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
console.warn(`⚠️ Warning: ESLint encountered issues: ${errorMessage}`);
|
||||
}
|
||||
|
||||
console.log('🧹 Running Prettier on generated/modified files...');
|
||||
try {
|
||||
// '--ignore-path' is necessary so the gitignored files ('local/' folder) can still be formatted
|
||||
execSync(`yarn prettier --write ${filesList} --ignore-path=./.prettierignore`, { cwd: basePath });
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
console.warn(`⚠️ Warning: Prettier encountered issues: ${errorMessage}`);
|
||||
}
|
||||
|
||||
return '✅ Files linted and formatted successfully!';
|
||||
} catch (error) {
|
||||
const errorMessage = error instanceof Error ? error.message : String(error);
|
||||
console.error('⚠️ Warning: Formatting operations failed:', errorMessage);
|
||||
return '⚠️ Warning: Formatting operations failed.';
|
||||
}
|
||||
};
|
||||
|
||||
export const validateGroup = (group: string) => {
|
||||
return group && group.includes('.grafana.app') ? true : 'Group should be in format: name.grafana.app';
|
||||
};
|
||||
|
||||
export const validateVersion = (version: string) => {
|
||||
return version && /^v\d+[a-z]*\d+$/.test(version) ? true : 'Version should be in format: v0alpha1, v1beta2, etc.';
|
||||
};
|
||||
@@ -0,0 +1,167 @@
|
||||
import path from 'path';
|
||||
import type { NodePlopAPI, PlopGeneratorConfig } from 'plop';
|
||||
|
||||
import {
|
||||
formatEndpoints,
|
||||
validateGroup,
|
||||
validateVersion,
|
||||
getFilesToFormat,
|
||||
runGenerateApis,
|
||||
formatFiles,
|
||||
// The file extension is necessary to make the imports
|
||||
// work with the '--experimental-strip-types' flag
|
||||
// @ts-ignore
|
||||
} from './helpers.ts';
|
||||
// @ts-ignore
|
||||
import { type ActionConfig, type PlopData, isPlopData } from './types.ts';
|
||||
|
||||
export default function plopGenerator(plop: NodePlopAPI) {
|
||||
// Grafana root path
|
||||
const basePath = path.resolve(import.meta.dirname, '../..');
|
||||
|
||||
// Register custom action types
|
||||
plop.setActionType('runGenerateApis', runGenerateApis(basePath));
|
||||
plop.setActionType('formatFiles', formatFiles(basePath));
|
||||
|
||||
// Used in templates to format endpoints
|
||||
plop.setHelper('formatEndpoints', formatEndpoints());
|
||||
|
||||
const generateRtkApiActions = (data: PlopData) => {
|
||||
const { reducerPath, groupName, version, isEnterprise } = data;
|
||||
|
||||
const apiClientBasePath = isEnterprise ? 'public/app/extensions/api/clients' : 'public/app/api/clients';
|
||||
const generateScriptPath = isEnterprise ? 'local/generate-enterprise-apis.ts' : 'scripts/generate-rtk-apis.ts';
|
||||
|
||||
// Using app path, so the imports work on any file level
|
||||
const clientImportPath = isEnterprise ? '../extensions/api/clients' : 'app/api/clients';
|
||||
|
||||
const apiPathPrefix = isEnterprise ? '../public/app/extensions/api/clients' : '../public/app/api/clients';
|
||||
|
||||
const templateData = {
|
||||
...data,
|
||||
apiPathPrefix,
|
||||
};
|
||||
|
||||
// Base actions that are always added
|
||||
const actions: ActionConfig[] = [
|
||||
{
|
||||
type: 'add',
|
||||
path: path.join(basePath, `${apiClientBasePath}/${groupName}/${version}/baseAPI.ts`),
|
||||
templateFile: './templates/baseAPI.ts.hbs',
|
||||
},
|
||||
{
|
||||
type: 'modify',
|
||||
path: path.join(basePath, generateScriptPath),
|
||||
pattern: '// PLOP_INJECT_API_CLIENT - Used by the API client generator',
|
||||
templateFile: './templates/config-entry.hbs',
|
||||
data: templateData,
|
||||
},
|
||||
{
|
||||
type: 'add',
|
||||
path: path.join(basePath, `${apiClientBasePath}/${groupName}/${version}/index.ts`),
|
||||
templateFile: './templates/index.ts.hbs',
|
||||
},
|
||||
];
|
||||
|
||||
// Only add redux reducer and middleware for OSS clients
|
||||
if (!isEnterprise) {
|
||||
actions.push(
|
||||
{
|
||||
type: 'modify',
|
||||
path: path.join(basePath, 'public/app/core/reducers/root.ts'),
|
||||
pattern: '// PLOP_INJECT_IMPORT',
|
||||
template: `import { ${reducerPath} } from '${clientImportPath}/${groupName}/${version}';\n// PLOP_INJECT_IMPORT`,
|
||||
},
|
||||
{
|
||||
type: 'modify',
|
||||
path: path.join(basePath, 'public/app/core/reducers/root.ts'),
|
||||
pattern: '// PLOP_INJECT_REDUCER',
|
||||
template: `[${reducerPath}.reducerPath]: ${reducerPath}.reducer,\n // PLOP_INJECT_REDUCER`,
|
||||
},
|
||||
{
|
||||
type: 'modify',
|
||||
path: path.join(basePath, 'public/app/store/configureStore.ts'),
|
||||
pattern: '// PLOP_INJECT_IMPORT',
|
||||
template: `import { ${reducerPath} } from '${clientImportPath}/${groupName}/${version}';\n// PLOP_INJECT_IMPORT`,
|
||||
},
|
||||
{
|
||||
type: 'modify',
|
||||
path: path.join(basePath, 'public/app/store/configureStore.ts'),
|
||||
pattern: '// PLOP_INJECT_MIDDLEWARE',
|
||||
template: `${reducerPath}.middleware,\n // PLOP_INJECT_MIDDLEWARE`,
|
||||
}
|
||||
);
|
||||
}
|
||||
|
||||
// Add formatting and generation actions
|
||||
actions.push(
|
||||
{
|
||||
type: 'formatFiles',
|
||||
files: getFilesToFormat(groupName, version, isEnterprise),
|
||||
},
|
||||
{
|
||||
type: 'runGenerateApis',
|
||||
isEnterprise,
|
||||
}
|
||||
);
|
||||
|
||||
return actions;
|
||||
};
|
||||
|
||||
const generator: PlopGeneratorConfig = {
|
||||
description: 'Generate RTK Query API client for a Grafana API group',
|
||||
prompts: [
|
||||
{
|
||||
type: 'confirm',
|
||||
name: 'isEnterprise',
|
||||
message: 'Is this a Grafana Enterprise API?',
|
||||
default: false,
|
||||
},
|
||||
{
|
||||
type: 'input',
|
||||
name: 'groupName',
|
||||
message: 'API group name (e.g. dashboard):',
|
||||
validate: (input: string) => (input?.trim() ? true : 'Group name is required'),
|
||||
},
|
||||
{
|
||||
type: 'input',
|
||||
name: 'group',
|
||||
message: 'API group (e.g. dashboard.grafana.app):',
|
||||
default: (answers: { groupName?: string }) => `${answers.groupName}.grafana.app`,
|
||||
validate: validateGroup,
|
||||
},
|
||||
{
|
||||
type: 'input',
|
||||
name: 'version',
|
||||
message: 'API version (e.g. v0alpha1):',
|
||||
default: 'v0alpha1',
|
||||
validate: validateVersion,
|
||||
},
|
||||
{
|
||||
type: 'input',
|
||||
name: 'reducerPath',
|
||||
message: 'Reducer path (e.g. dashboardAPIv0alpha1):',
|
||||
default: (answers: { groupName?: string; version?: string }) => `${answers.groupName}API${answers.version}`,
|
||||
validate: (input: string) =>
|
||||
input?.endsWith('API') || input?.match(/API[a-z]\d+[a-z]*\d*$/)
|
||||
? true
|
||||
: 'Reducer path should end with "API" or "API<version>" (e.g. dashboardAPI, dashboardAPIv0alpha1)',
|
||||
},
|
||||
{
|
||||
type: 'input',
|
||||
name: 'endpoints',
|
||||
message: 'Endpoints to include (comma-separated, optional):',
|
||||
validate: () => true,
|
||||
},
|
||||
],
|
||||
actions: function (data) {
|
||||
if (!isPlopData(data)) {
|
||||
throw new Error('Invalid data format received from prompts');
|
||||
}
|
||||
|
||||
return generateRtkApiActions(data);
|
||||
},
|
||||
};
|
||||
|
||||
plop.setGenerator('rtk-api-client', generator);
|
||||
}
|
||||
@@ -0,0 +1,14 @@
|
||||
import { createApi } from '@reduxjs/toolkit/query/react';
|
||||
|
||||
import { createBaseQuery } from 'app/api/createBaseQuery';
|
||||
import { getAPIBaseURL } from 'app/api/utils';
|
||||
|
||||
export const BASE_URL = getAPIBaseURL('{{group}}', '{{version}}');
|
||||
|
||||
export const api = createApi({
|
||||
reducerPath: '{{reducerPath}}',
|
||||
baseQuery: createBaseQuery({
|
||||
baseURL: BASE_URL,
|
||||
}),
|
||||
endpoints: () => ({}),
|
||||
});
|
||||
@@ -0,0 +1,9 @@
|
||||
'{{apiPathPrefix}}/{{groupName}}/{{version}}/endpoints.gen.ts': {
|
||||
apiFile: '{{apiPathPrefix}}/{{groupName}}/{{version}}/baseAPI.ts',
|
||||
schemaFile: '../data/openapi/{{group}}-{{version}}.json',
|
||||
{{#if endpoints}}
|
||||
filterEndpoints: [{{{formatEndpoints endpoints}}}],
|
||||
{{/if}}
|
||||
tag: true,
|
||||
},
|
||||
// PLOP_INJECT_API_CLIENT - Used by the API client generator
|
||||
@@ -0,0 +1,3 @@
|
||||
import { generatedAPI } from './endpoints.gen';
|
||||
|
||||
export const {{reducerPath}} = generatedAPI.enhanceEndpoints({});
|
||||
@@ -0,0 +1,27 @@
|
||||
import type { AddActionConfig, ModifyActionConfig } from 'plop';
|
||||
|
||||
export interface FormatFilesActionConfig {
|
||||
type: 'formatFiles';
|
||||
files: string[];
|
||||
}
|
||||
|
||||
export interface RunGenerateApisActionConfig {
|
||||
type: 'runGenerateApis';
|
||||
isEnterprise: boolean;
|
||||
}
|
||||
|
||||
// Union type of all possible action configs
|
||||
export type ActionConfig = AddActionConfig | ModifyActionConfig | FormatFilesActionConfig | RunGenerateApisActionConfig;
|
||||
|
||||
export interface PlopData {
|
||||
groupName: string;
|
||||
group: string;
|
||||
version: string;
|
||||
reducerPath: string;
|
||||
endpoints: string;
|
||||
isEnterprise: boolean;
|
||||
}
|
||||
|
||||
export function isPlopData(data: unknown): data is PlopData {
|
||||
return typeof data === 'object' && data !== null;
|
||||
}
|
||||
@@ -9,72 +9,10 @@ for file in "$ARTIFACTS_DIR"/*.tgz; do
|
||||
echo "🔍 Checking NPM package: $file"
|
||||
|
||||
# Ignore named-exports for now as builds aren't compatible yet.
|
||||
yarn dlx @arethetypeswrong/cli "$file" --ignore-rules "named-exports"
|
||||
|
||||
# get filename then strip everything after package name.
|
||||
dir_name=$(basename "$file" .tgz | sed -E 's/@([a-zA-Z0-9-]+)-[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9-]+)?/\1/')
|
||||
mkdir -p "./npm-artifacts/$dir_name"
|
||||
tar -xzf "$file" -C "./npm-artifacts/$dir_name" --strip-components=1
|
||||
|
||||
# Make sure the tar wasn't empty
|
||||
if [ ! -d "./npm-artifacts/$dir_name" ]; then
|
||||
echo -e "❌ Failed: Empty package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Navigate inside the new extracted directory
|
||||
pushd "./npm-artifacts/$dir_name" || exit
|
||||
|
||||
# Check for required files
|
||||
check_files=("package.json" "README.md" "CHANGELOG.md")
|
||||
for check_file in "${check_files[@]}"; do
|
||||
if [ ! -f "$check_file" ]; then
|
||||
echo -e "❌ Failed: Missing required file $check_file in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Check license files
|
||||
if [ -f "LICENSE_APACHE2" ] || [ -f "LICENSE_AGPL" ]; then
|
||||
echo -e "Found required license file in package $dir_name.\n"
|
||||
else
|
||||
echo -e "❌ Failed: Missing required license file in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Assert commonjs builds
|
||||
if [ ! -d dist ] || [ ! -f dist/cjs/index.cjs ] || [ ! -f dist/cjs/index.d.cts ]; then
|
||||
echo -e "❌ Failed: Missing 'dist' directory or required commonjs files in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(jq -r '.main' package.json)" != "./dist/cjs/index.cjs" ] || \
|
||||
[ "$(jq -r '.types' package.json)" != "./dist/cjs/index.d.cts" ]; then
|
||||
echo -e "❌ Failed: Incorrect cjs package.json properties in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Assert esm builds
|
||||
esm_packages=("grafana-data" "grafana-ui" "grafana-runtime" "grafana-e2e-selectors" "grafana-schema")
|
||||
for esm_package in "${esm_packages[@]}"; do
|
||||
if [[ "$dir_name" == "$esm_package" ]]; then
|
||||
if [ ! -d dist/esm ] || [ ! -f dist/esm/index.mjs ]; then
|
||||
echo -e "❌ Failed: Missing 'dist/esm' directory or required esm files in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [ "$(jq -r '.module' package.json)" != "./dist/esm/index.mjs" ]; then
|
||||
echo -e "❌ Failed: Incorrect esm package.json properties in package $dir_name.\n"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo -e "✅ Passed: package checks for $file.\n"
|
||||
popd || exit
|
||||
yarn attw "$file" --ignore-rules "named-exports"
|
||||
yarn publint "$file"
|
||||
|
||||
done
|
||||
|
||||
echo "🚀 All NPM package checks passed! 🚀"
|
||||
rm -rf "${ARTIFACTS_DIR:?}/"*/
|
||||
exit 0
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
const CopyWebpackPlugin = require('copy-webpack-plugin');
|
||||
const path = require('path');
|
||||
const webpack = require('webpack');
|
||||
|
||||
@@ -68,6 +69,14 @@ module.exports = {
|
||||
new webpack.ProvidePlugin({
|
||||
Buffer: ['buffer', 'Buffer'],
|
||||
}),
|
||||
new CopyWebpackPlugin({
|
||||
patterns: [
|
||||
{
|
||||
from: 'public/img',
|
||||
to: 'img',
|
||||
},
|
||||
],
|
||||
}),
|
||||
],
|
||||
module: {
|
||||
rules: [
|
||||
|
||||
@@ -4,6 +4,7 @@ const browserslist = require('browserslist');
|
||||
const { resolveToEsbuildTarget } = require('esbuild-plugin-browserslist');
|
||||
const ESLintPlugin = require('eslint-webpack-plugin');
|
||||
const ForkTsCheckerWebpackPlugin = require('fork-ts-checker-webpack-plugin');
|
||||
const fs = require('fs');
|
||||
const MiniCssExtractPlugin = require('mini-css-extract-plugin');
|
||||
const path = require('path');
|
||||
const { DefinePlugin, EnvironmentPlugin } = require('webpack');
|
||||
@@ -29,6 +30,21 @@ function getDecoupledPlugins() {
|
||||
return packages.filter((pkg) => pkg.dir.includes('plugins/datasource')).map((pkg) => `${pkg.dir}/**`);
|
||||
}
|
||||
|
||||
// When linking scenes for development, resolve the path to the src directory for sourcemaps
|
||||
function scenesModule() {
|
||||
const scenesPath = path.resolve('./node_modules/@grafana/scenes');
|
||||
try {
|
||||
const status = fs.lstatSync(scenesPath);
|
||||
if (status.isSymbolicLink()) {
|
||||
console.log(`scenes is linked to local scenes repo`);
|
||||
return path.resolve(scenesPath + '/src');
|
||||
}
|
||||
} catch (error) {
|
||||
console.error(`Error checking scenes path: ${error.message}`);
|
||||
}
|
||||
return scenesPath;
|
||||
}
|
||||
|
||||
const envConfig = getEnvConfig();
|
||||
|
||||
module.exports = (env = {}) => {
|
||||
@@ -55,6 +71,7 @@ module.exports = (env = {}) => {
|
||||
// This is required to correctly resolve react-router-dom when linking with
|
||||
// local version of @grafana/scenes
|
||||
'react-router-dom': path.resolve('./node_modules/react-router-dom'),
|
||||
'@grafana/scenes': scenesModule(),
|
||||
},
|
||||
},
|
||||
|
||||
|
||||
Reference in New Issue
Block a user