CI Migration: Backport pkg/build (daggerbuild) and GHAs to release-12.0.3 (#107997)
* CI: move grafana-build into pkg/build (#105640) * move grafana-build into pkg/build * Update go.mod/go.sum files after cherry-pick build - Updates from workspace sync after cherry-picking daggerbuild package - Resolves dependency conflicts from pkg/build integration - Required for successful build after CI migration backport * CI Migration: Replace .github with main branch version - Complete replacement of .github directory from main branch - Includes updated workflows, actions, and configuration files - Ensures all CI improvements from main are included - Next step: fix branch-specific issues in separate commits * Fix branch triggers: Update workflows to target release branches - Update 8 workflows to trigger on release-* instead of main - Ensures workflows run on pull requests to release branches - Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator * Fix branch triggers: Update workflows to target both main and release branches - CORRECTED: Include both main and release-* branch patterns - Ensures workflows run on both main branch (original functionality) and release branches (needed for backport) - Previous commit incorrectly removed main branch support - Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator * CI Migration: Update .gitignore to allow OSS wire file to be committed - Remove **/wire_gen.go (allow pkg/server/wire_gen.go to be committed) - Add /pkg/server/enterprise_wire_gen.go (keep enterprise wire file ignored) - This enables the new committed wire files architecture for CI * CI Migration: Add enhanced wire tool with automatic build constraints - Backport enhanced wire tool from main branch with -gen_tags flag - This enables automatic regeneration with proper constraints to prevent conflicts - Wire file contains all required functions: Initialize, InitializeDocumentBuilders, etc. - Solves the missing wire functions issue causing CI failures - Enhanced tool ensures build constraints persist on regeneration * CI Migration: Update Makefile to use enhanced wire tool with build constraints - Update gen-go target to use -gen_tags flag for automatic build constraints - Ensures future engineers get proper build constraints when running 'make gen-go' - Matches main branch approach for consistent wire file generation - Removes dependency on WIRE_TAGS variable and GO_RACE_FLAG for simplicity * Add gen-enterprise-go target to Makefile for release-12.0.3 - Backports gen-enterprise-go target from main branch - Enables enterprise wire file generation in release branches - Part of CI migration backport wire infrastructure fixes - Resolves CI issues where enterprise wire functions were missing * Fix wire tool golden file format for Drone CI compatibility - Update all 43 golden test files to match main branch format - Change go:generate command from github.com/google/wire/cmd/wire to ./pkg/build/wire/cmd/wire/main.go - Remove legacy '// +build !wireinject' constraints - Fixes Drone CI test failures that were blocking OSS PR merge - All wire tests now pass successfully This aligns the backport branch with the golden file format updates that were made in main branch when the wire tool was enhanced. * Update swagger specs to include enterprise APIs - Regenerated api-enterprise-spec.json with enterprise API definitions - Updated api-merged.json with merged OSS and enterprise specs - Regenerated openapi3.json with complete API specifications - Fixes enterprise CI swagger generation validation failures - Enterprise APIs now properly detected and documented * Fix OSS integration tests missing enterprise build tags in Drone CI - Add -tags=enterprise to all integration test steps in lib.star - Fixes test-backend-integration, postgres-integration-tests, and mysql-integration-tests - Regenerate .drone.yml with proper enterprise build tags - Resolves 'server could not find the requested resource' errors for enterprise APIs - Ensures enterprise APIs (querylibrary, reporting, banners, scim) are registered during tests - Aligns OSS Drone CI with GitHub Actions behavior and enterprise repository * Fix CODEOWNERS for release-12.0.3 compatibility - Remove 21 entries that reference files/directories not present in release branch - Resolves File Exist Checker failures in codeowners-validator workflow - Maintains team ownership assignments from main branch for existing files - Lines removed: .air.toml, apps/secret/, apps/iam/, e2e-playwright/, packages/grafana-alerting/, etc. * baldm0mma/ update releasefinder * Add i18n-extract script for release branch compatibility - Add 'i18n-extract': 'make i18n-extract' to package.json scripts - Resolves i18n-verify workflow failure that expects this script - The make target already exists and handles OSS + enterprise i18n extraction - Maintains i18n verification functionality from main branch workflows - Simple safe addition that calls existing make infrastructure * Phase 4: Fix GitHub Actions workflow branch triggers - Add release-*.*.* pattern to 8 workflows missing it - Fix duplicate release branch patterns in 3 workflows - Ensure consistent branch trigger format across all workflows - Critical workflows now properly trigger on release branches Fixes workflows: actionlint, backend-code-checks, go-lint, reject-gh-secrets, run-schema-v2-e2e, shellcheck, swagger-gen, trivy-scan * Fix: Backport E2E runner infrastructure for Enterprise CI - Add e2e/main.go and e2e/internal/ directory from main branch - Add urfave/cli/v3 dependency required by E2E runner - Fixes Enterprise CI failure: 'Build E2E test runner' and 'Build & package Grafana for e2e tests' - Root cause: E2E runner infrastructure was added to main after release-12.0.3 branch creation - Solution: Backport missing E2E runner files to enable Enterprise CI completion Resolves: no Go files in /opt/actions-runner/_work/grafana-enterprise/grafana-enterprise/grafana/e2e * Fix: Update go.mod dependency classification for urfave/cli/v3 - Change urfave/cli/v3 from indirect to direct dependency - Fixes Go Workspace Check failure in CI - Required after adding E2E runner infrastructure that directly imports urfave/cli/v3 - Resolves: make update-workspace corrects dependency classification * Fix: Add team ownership for urfave/cli/v3 dependency - Assign @grafana/grafana-backend-group as owner for github.com/urfave/cli/v3@v3.3.8 - Follows existing pattern: urfave/cli v1 and v2 also owned by grafana-backend-group - Resolves Backend Code Checks / Validate Backend Configs CI failure - Required for E2E runner infrastructure dependency ownership compliance Fixes: modowners check requiring team assignment for newly added dependencies * Fix: Revert experimental E2E playwright infrastructure to stable version - Revert pr-e2e-tests.yml to stable Cypress-based E2E testing - Remove experimental storybook-verification-playwright.yml workflow - Revert run-dashboard-search-e2e.yml and release-pr.yml to use e2e/test-plugins/ - Keep stable E2E runner infrastructure (e2e/main.go + e2e/internal/) - Remove experimental playwright features per team recommendation Team feedback: Playwright tests are experimental and shouldn't be backported to stable release branches * Fix: Complete cleanup of experimental playwright dependencies - Revert package.json to stable version (remove e2e-playwright scripts and path references) - Revert playwright.config.ts to stable plugin-e2e configuration - Remove all experimental playwright infrastructure dependencies - Ensure clean stable E2E testing environment All experimental features removed per team recommendation for stable release branches * Fix: Restore working package.json configuration - Revert package.json to version 12.0.3 (working release branch version) - Fix workspace dependency resolution issues caused by incorrect revert to main branch version - FE tests, betterer, and linting should now work correctly Issue was caused by reverting package.json to main branch (12.1.0-pre) instead of keeping the working release branch configuration (12.0.3) * CI: mirror some CI dependencies (#106148) * mirror some CI dependencies * remove -v from go build * Fix: Backport missing e2e/run-suite script for daggerbuild E2E system The daggerbuild E2E system (used by OSS workflows) expects ./e2e/run-suite to exist, but we only backported the new E2E runner infrastructure. This script is needed for: - OSS workflow: 'go run ./pkg/build/e2e --suite=dashboards-suite' - Legacy Cypress configuration with video support - Integration between daggerbuild and existing Cypress test suites Resolves E2E test failures in OSS workflows where videos directory cannot be found because run-suite script was missing. * Fix: Resolve daggerbuild E2E path doubling issue - Extract just suite name from full path using filepath.Base() - Prevents doubled paths like './e2e/e2e/dashboards-suite/videos' - Resolves 'no spec files found' and 'no such file or directory' errors - GitHub Actions passes full paths like 'e2e/dashboards-suite' but run-suite script expects just 'dashboards-suite' * Infrastructure: Wholesale copy pkg/build/ from main COMPLETE DAGGERBUILD SYSTEM UPDATE: - Modern E2E system with CLI framework and --flags support - New accessibility testing system (a11y/) - New Playwright E2E testing system (e2e-playwright/) - External infrastructure improvements (musl.cc → dl.grafana.com/ci) - Updated daggerbuild core components with latest fixes - Updated Go dependencies and wire modules ARCHITECTURAL COMPATIBILITY VERIFIED: - Modern pkg/build calls: ./e2e-runner cypress --start-grafana=false --cypress-video - Our e2e runner supports: All required parameters including --suite, --env flags - GitHub Actions workflows: Pass compatible arguments - Binary builds: Successfully creates working e2e-runner RESOLVES ISSUES: - Fixes --flags parameter compatibility with GitHub Actions - Includes all follow-up infrastructure improvements - Provides complete, tested system with modern CLI framework - Eliminates external dependency failures (musl.cc timeouts) - Removes need for path doubling workarounds (modern system handles full paths correctly) Replaces incremental cherry-picking with complete, tested system from main. * Dependencies: Update Go modules after pkg/build wholesale copy DEPENDENCY UPDATES: - Updated go.work.sum with new dependencies from modern pkg/build system - Updated all workspace module dependencies (go.mod/go.sum files) - Removed pkg/build/cmd/enterprise.go (enterprise-only file, gets copied during enterprise development) ENTERPRISE INTEGRATION FIX: - enterprise.go file doesn't belong in OSS repository - Gets copied from grafana-enterprise during development mode - Main branch doesn't have this file, explaining module compatibility All Go modules now properly resolved and compatible with modern pkg/build architecture. * Dependencies: Update workspace Go module checksums after pkg/build wholesale copy - Synchronizes all go.sum files across workspace modules - Adds missing .mod.h1 entries that were required after infrastructure update - Resolves Go Workspace Check CI failures - Updates 22 modules: .citools/, apps/, pkg/ subdirectories plus go.work.sum Addresses CI error: 'Please run make update-workspace and commit the changes' * Infrastructure: Revert to pre-Playwright E2E system for release branch compatibility Strategic combination of modern daggerbuild with stable E2E infrastructure: **Modern Daggerbuild (from main):** - Latest CLI framework (github.com/urfave/cli/v3) - External infrastructure fixes (musl.cc → dl.grafana.com/ci) - Complete pkg/build/ system with all enhancements - Updated dependencies and architecture improvements **Stable E2E Infrastructure (pre-Playwright):** - pkg/build/e2e/service.go: Reverted to version before commitb6580ccb10- pkg/build/a11y/: Reverted to stable version compatible with existing infrastructure - Removed experimental pkg/build/e2e-playwright/ system - Removed experimental e2e-playwright/ test infrastructure **Why This Approach:** - GitHub Actions workflows call 'go run ./pkg/build/e2e' (modern system) - Modern system now expects only standard e2e/ directory (compatible) - Release branches have standard e2e/ infrastructure (dashboards-suite, various-suite, etc.) - Eliminates dependency on experimental Playwright infrastructure **Key Dependencies Met:** - e2e/pa11yci.conf.js: Added for a11y test compatibility - /src/e2e directory: Standard directory structure (no e2e-playwright needed) - Existing test plugins: Uses stable e2e/test-plugins/ **Result:** Best of both worlds - modern, reliable daggerbuild infrastructure with proven E2E testing that works across all release branches without requiring experimental dependencies. * Fix: Remove unused embed.go file causing golangci-lint failure - Removes pkg/build/daggerbuild/msi/embed.go with unused embed.FS variable - MSI build process uses directory mounting instead of embedded filesystem - Resolves 'var resources is unused (unused)' linting error * Fix: Update Node.js version to match working E2E configuration - Changes from v22.11.0 to v22.16.0 to match commit17952d45e4- Working commit had all E2E tests passing with same daggerbuild infrastructure - Node.js version differences can affect frontend builds and UI rendering - Ensures consistent environment between working reference and current branch * Fix: Use conditional container base for E2E compatibility - Alpine for OSS tests (better old arch dashboardScene=false compatibility) - Ubuntu for Enterprise tests (when image renderer needed) - Restores original release-12.0.3 Alpine behavior for OSS - Should resolve old arch E2E test failures * Fix: Go fmt formatting for conditional container logic * Revert: Remove unnecessary Alpine/Ubuntu conditional logic - Real issue was wrong environment variable name (dashboardScene vs DISABLE_SCENES) - Keep Ubuntu base (original main branch behavior) for simplicity - Container base was not the root cause of old arch failures * Experiment: Copy scripts/ from pre-Playwright commit (ea0ddb3fc9) - Copied scripts directory from commitea0ddb3fc9(just before Playwright migration) - This overwrites some CI migration fixes but testing if it's actually needed - Can revert back tob9b4602dbdif CI breaks Test commit to see real impact on GitHub Actions workflows. * trigger ci 01 * Fix: Copy .drone.star from main to match scripts structure - Resolves Starlark evaluation error for missing integration_test_pipelines - Aligns .drone.star with main branch scripts structure - Same fix applied to both OSS and Enterprise repositories * make drone * Fix ARM/v7 Docker build failures by using dagger run for cross-compilation - Change artifacts_cmd() in scripts/drone/steps/rgm.star to use 'dagger run go run' instead of 'go run' - Add --max-execution-steps 100000 to make drone target in Makefile to properly regenerate .drone.yml - This fixes CGO cross-compilation issues with SQLite for ARM/v7 Docker builds - Matches the approach already used successfully in Enterprise builds * Fix Docker base image references in rgm build steps - Change from symbolic names (alpine-base, ubuntu-base) to actual Docker images - alpine-base → alpine:3.21.3 - ubuntu-base → ubuntu:22.04 - Add ubuntu and alpine parameters to rgm_artifacts_step and rgm_build_docker_step functions - Fixes Docker pull errors: 'repository does not exist or may require authorization' - Aligns OSS configuration with Enterprise approach --------- Co-authored-by: Kevin Minehart <5140827+kminehart@users.noreply.github.com>
This commit is contained in:
Jev Forsberg
co-authored by
Kevin Minehart
Kevin Minehart
parent
aee6b20c7f
commit
dc6bd2a20a
@@ -0,0 +1,102 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/docker"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var (
|
||||
DockerRegistryFlag = &cli.StringFlag{
|
||||
Name: "registry",
|
||||
Usage: "Prefix the image name with the registry provided",
|
||||
Value: "docker.io",
|
||||
}
|
||||
DockerOrgFlag = &cli.StringFlag{
|
||||
Name: "org",
|
||||
Usage: "Overrides the organization of the images",
|
||||
Value: "grafana",
|
||||
}
|
||||
AlpineImageFlag = &cli.StringFlag{
|
||||
Name: "alpine-base",
|
||||
Usage: "The image or image alias specified in the Dockerfile to be used as the base image when building the Alpine version of the Grafana docker image.",
|
||||
Value: "alpine-base",
|
||||
}
|
||||
UbuntuImageFlag = &cli.StringFlag{
|
||||
Name: "ubuntu-base",
|
||||
Usage: "The image or image alias specified in the Dockerfile to be used as the base image when building the Ubuntu version of the Grafana docker image",
|
||||
Value: "ubuntu-base",
|
||||
}
|
||||
TagFormatFlag = &cli.StringFlag{
|
||||
Name: "tag-format",
|
||||
Usage: "Provide a go template for formatting the docker tag(s) for images with an Alpine base",
|
||||
Value: docker.DefaultTagFormat,
|
||||
}
|
||||
UbuntuTagFormatFlag = &cli.StringFlag{
|
||||
Name: "ubuntu-tag-format",
|
||||
Usage: "Provide a go template for formatting the docker tag(s) for images with a ubuntu base",
|
||||
Value: docker.DefaultUbuntuTagFormat,
|
||||
}
|
||||
BoringTagFormatFlag = &cli.StringFlag{
|
||||
Name: "boring-tag-format",
|
||||
Usage: "Provide a go template for formatting the docker tag(s) for the boringcrypto build of Grafana Enterprise",
|
||||
Value: docker.DefaultBoringTagFormat,
|
||||
}
|
||||
|
||||
ProDockerRegistryFlag = &cli.StringFlag{
|
||||
Name: "pro-registry",
|
||||
Usage: "Prefix the image name with the registry provided",
|
||||
Value: "docker.io",
|
||||
}
|
||||
ProDockerOrgFlag = &cli.StringFlag{
|
||||
Name: "pro-org",
|
||||
Usage: "Overrides the organization of the images",
|
||||
Value: "grafana",
|
||||
}
|
||||
ProDockerRepoFlag = &cli.StringFlag{
|
||||
Name: "pro-repo",
|
||||
Usage: "Overrides the docker repository of the built images",
|
||||
Value: "grafana-pro",
|
||||
}
|
||||
|
||||
EntDockerRegistryFlag = &cli.StringFlag{
|
||||
Name: "docker-enterprise-registry",
|
||||
Usage: "Prefix the image name with the registry provided",
|
||||
Value: "docker.io",
|
||||
}
|
||||
EntDockerOrgFlag = &cli.StringFlag{
|
||||
Name: "docker-enterprise-org",
|
||||
Usage: "Overrides the organization of the images",
|
||||
Value: "grafana",
|
||||
}
|
||||
EntDockerRepoFlag = &cli.StringFlag{
|
||||
Name: "docker-enterprise-repo",
|
||||
Usage: "Overrides the docker repository of the built images",
|
||||
Value: "grafana-enterprise",
|
||||
}
|
||||
|
||||
HGTagFormatFlag = &cli.StringFlag{
|
||||
Name: "hg-tag-format",
|
||||
Usage: "Provide a go template for formatting the docker tag(s) for Hosted Grafana images",
|
||||
Value: docker.DefaultHGTagFormat,
|
||||
}
|
||||
|
||||
DockerRegistry = pipeline.NewStringFlagArgument(DockerRegistryFlag)
|
||||
DockerOrg = pipeline.NewStringFlagArgument(DockerOrgFlag)
|
||||
AlpineImage = pipeline.NewStringFlagArgument(AlpineImageFlag)
|
||||
UbuntuImage = pipeline.NewStringFlagArgument(UbuntuImageFlag)
|
||||
TagFormat = pipeline.NewStringFlagArgument(TagFormatFlag)
|
||||
UbuntuTagFormat = pipeline.NewStringFlagArgument(UbuntuTagFormatFlag)
|
||||
BoringTagFormat = pipeline.NewStringFlagArgument(BoringTagFormatFlag)
|
||||
|
||||
// The docker registry for Grafana Pro is often different than the one for Grafana & Enterprise
|
||||
ProDockerRegistry = pipeline.NewStringFlagArgument(ProDockerRegistryFlag)
|
||||
ProDockerOrg = pipeline.NewStringFlagArgument(ProDockerOrgFlag)
|
||||
ProDockerRepo = pipeline.NewStringFlagArgument(ProDockerRepoFlag)
|
||||
|
||||
EntDockerRegistry = pipeline.NewStringFlagArgument(EntDockerRegistryFlag)
|
||||
EntDockerOrg = pipeline.NewStringFlagArgument(EntDockerOrgFlag)
|
||||
EntDockerRepo = pipeline.NewStringFlagArgument(EntDockerRepoFlag)
|
||||
|
||||
HGTagFormat = pipeline.NewStringFlagArgument(HGTagFormatFlag)
|
||||
)
|
||||
@@ -0,0 +1,4 @@
|
||||
// Package arguments holds globally-defined arguments that are used throughout the program for shared data.
|
||||
// A good candidate for an argument is a directory whose contents that may be used in the creation of multiple artifacts, like the Grafana source directory.
|
||||
// Arguments are different than flags; a flag is a boolean argument in an artifact string which can set one or multiple preset values.
|
||||
package arguments
|
||||
@@ -0,0 +1 @@
|
||||
package arguments
|
||||
@@ -0,0 +1,53 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
|
||||
"dagger.io/dagger"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/golang"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var GoBuildCache = pipeline.Argument{
|
||||
Name: "go-cache-volume",
|
||||
Description: "Mounted at GOCACHE when building Go backends",
|
||||
ArgumentType: pipeline.ArgumentTypeCacheVolume,
|
||||
Flags: []cli.Flag{},
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
return opts.Client.CacheVolume("go-build-cache"), nil
|
||||
},
|
||||
}
|
||||
|
||||
var GoModCache = pipeline.Argument{
|
||||
Name: "go-mod-volume",
|
||||
Description: "Stores downloaded Go modules when building Go backends",
|
||||
ArgumentType: pipeline.ArgumentTypeCacheVolume,
|
||||
Flags: []cli.Flag{},
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
vol := opts.Client.CacheVolume("go-mod-cache")
|
||||
goVersion, err := opts.State.String(ctx, GoVersion)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
src, err := opts.State.Directory(ctx, GrafanaDirectory)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
c := golang.Container(opts.Client, opts.Platform, goVersion).
|
||||
WithEnvVariable("GOMODCACHE", "/go/pkg/mod").
|
||||
WithMountedCache("/go/pkg/mod", vol).
|
||||
WithDirectory("/src", src, dagger.ContainerWithDirectoryOpts{
|
||||
Include: []string{"**/*.mod", "**/*.sum", "**/*.work"},
|
||||
}).
|
||||
WithWorkdir("/src").
|
||||
WithExec([]string{"go", "mod", "download"})
|
||||
|
||||
if _, err := c.Sync(ctx); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return vol, nil
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
const (
|
||||
DefaultViceroyVersion = "v0.4.0"
|
||||
)
|
||||
|
||||
var GoVersion = pipeline.Argument{
|
||||
Name: "go-version",
|
||||
Description: "The Go version to use when compiling Grafana",
|
||||
ArgumentType: pipeline.ArgumentTypeString,
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
src, err := opts.State.Directory(ctx, GrafanaDirectory)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
stdout, err := opts.Client.Container().From("alpine").
|
||||
WithMountedFile("/src/go.mod", src.File("go.mod")).
|
||||
WithWorkdir("/src").
|
||||
WithExec([]string{"/bin/sh", "-c", `grep '^go ' go.mod | awk '{print $2}'`}).
|
||||
Stdout(ctx)
|
||||
|
||||
return strings.TrimSpace(stdout), err
|
||||
},
|
||||
}
|
||||
|
||||
var ViceroyVersionFlag = &cli.StringFlag{
|
||||
Name: "viceroy-version",
|
||||
Usage: "This flag sets the base image of the container used to build the Grafana backend binaries for non-Linux distributions",
|
||||
Value: DefaultViceroyVersion,
|
||||
}
|
||||
|
||||
var ViceroyVersion = pipeline.NewStringFlagArgument(ViceroyVersionFlag)
|
||||
@@ -0,0 +1,28 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var (
|
||||
GPGPublicKeyFlag = &cli.StringFlag{
|
||||
Name: "gpg-public-key-base64",
|
||||
Usage: "Provides a public key encoded in base64 for GPG signing",
|
||||
EnvVars: []string{"GPG_PUBLIC_KEY"},
|
||||
}
|
||||
GPGPrivateKeyFlag = &cli.StringFlag{
|
||||
Name: "gpg-private-key-base64",
|
||||
Usage: "Provides a private key encoded in base64 for GPG signing",
|
||||
EnvVars: []string{"GPG_PRIVATE_KEY"},
|
||||
}
|
||||
GPGPassphraseFlag = &cli.StringFlag{
|
||||
Name: "gpg-passphrase",
|
||||
Usage: "Provides a private key passphrase encoded in base64 for GPG signing",
|
||||
EnvVars: []string{"GPG_PASSPHRASE"},
|
||||
}
|
||||
|
||||
GPGPublicKey = pipeline.NewStringFlagArgument(GPGPublicKeyFlag)
|
||||
GPGPrivateKey = pipeline.NewStringFlagArgument(GPGPrivateKeyFlag)
|
||||
GPGPassphrase = pipeline.NewStringFlagArgument(GPGPassphraseFlag)
|
||||
)
|
||||
@@ -0,0 +1,299 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"path"
|
||||
"path/filepath"
|
||||
|
||||
"dagger.io/dagger"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/cliutil"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/containers"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/daggerutil"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/frontend"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/git"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
const BusyboxImage = "busybox:1.36"
|
||||
|
||||
func InitializeEnterprise(d *dagger.Client, grafana *dagger.Directory, enterprise *dagger.Directory) *dagger.Directory {
|
||||
hash := d.Container().From("alpine/git").
|
||||
WithDirectory("/src/grafana-enterprise", enterprise).
|
||||
WithWorkdir("/src/grafana-enterprise").
|
||||
WithEntrypoint([]string{}).
|
||||
WithExec([]string{"/bin/sh", "-c", "git rev-parse HEAD > .buildinfo.enterprise-commit"}).
|
||||
File("/src/grafana-enterprise/.buildinfo.enterprise-commit")
|
||||
|
||||
return d.Container().From(BusyboxImage).
|
||||
WithDirectory("/src/grafana", grafana).
|
||||
WithDirectory("/src/grafana-enterprise", enterprise).
|
||||
WithWorkdir("/src/grafana-enterprise").
|
||||
WithFile("/src/grafana/.buildinfo.enterprise-commit", hash).
|
||||
WithExec([]string{"/bin/sh", "build.sh"}).
|
||||
WithExec([]string{"cp", "LICENSE", "../grafana"}).
|
||||
Directory("/src/grafana")
|
||||
}
|
||||
|
||||
// GrafnaaOpts are populated by the 'GrafanaFlags' flags.
|
||||
// These options define how to mount or clone the grafana/enterprise source code.
|
||||
type GrafanaDirectoryOpts struct {
|
||||
// GrafanaDir is the path to the Grafana source tree.
|
||||
// If GrafanaDir is empty, then we're most likely cloning Grafana and using that as a directory.
|
||||
GrafanaDir string
|
||||
EnterpriseDir string
|
||||
// GrafanaRepo will clone Grafana from a different repository when cloning Grafana.
|
||||
GrafanaRepo string
|
||||
EnterpriseRepo string
|
||||
// GrafanaRef will checkout a specific tag, branch, or commit when cloning Grafana.
|
||||
GrafanaRef string
|
||||
EnterpriseRef string
|
||||
// GitHubToken is used when cloning Grafana/Grafana Enterprise.
|
||||
GitHubToken string
|
||||
|
||||
PatchesRepo string
|
||||
PatchesPath string
|
||||
PatchesRef string
|
||||
}
|
||||
|
||||
func githubToken(ctx context.Context, token string) (string, error) {
|
||||
// Since GrafanaDir was not provided, we must clone it.
|
||||
ght := token
|
||||
|
||||
// If GitHubToken was not set from flag
|
||||
if ght != "" {
|
||||
return ght, nil
|
||||
}
|
||||
|
||||
token, err := git.LookupGitHubToken(ctx)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if token == "" {
|
||||
return "", fmt.Errorf("unable to acquire github token")
|
||||
}
|
||||
|
||||
return token, nil
|
||||
}
|
||||
|
||||
func GrafanaDirectoryOptsFromFlags(c cliutil.CLIContext) *GrafanaDirectoryOpts {
|
||||
return &GrafanaDirectoryOpts{
|
||||
GrafanaRepo: c.String("grafana-repo"),
|
||||
EnterpriseRepo: c.String("enterprise-repo"),
|
||||
GrafanaDir: c.String("grafana-dir"),
|
||||
EnterpriseDir: c.String("enterprise-dir"),
|
||||
GrafanaRef: c.String("grafana-ref"),
|
||||
EnterpriseRef: c.String("enterprise-ref"),
|
||||
GitHubToken: c.String("github-token"),
|
||||
PatchesRepo: c.String("patches-repo"),
|
||||
PatchesPath: c.String("patches-path"),
|
||||
PatchesRef: c.String("patches-ref"),
|
||||
}
|
||||
}
|
||||
|
||||
func cloneOrMount(ctx context.Context, client *dagger.Client, localPath, repo, ref string, ght string) (*dagger.Directory, error) {
|
||||
if localPath != "" {
|
||||
absolute, err := filepath.Abs(localPath)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error getting absolute path for local dir: %w", err)
|
||||
}
|
||||
localPath = absolute
|
||||
slog.Info("Using local directory for repository", "path", localPath, "repo", repo)
|
||||
return daggerutil.HostDir(client, localPath)
|
||||
}
|
||||
|
||||
ght, err := githubToken(ctx, ght)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error acquiring GitHub token: %w", err)
|
||||
}
|
||||
|
||||
return git.CloneWithGitHubToken(client, ght, repo, ref)
|
||||
}
|
||||
|
||||
func applyPatches(ctx context.Context, client *dagger.Client, src *dagger.Directory, repo, patchesPath, ref, ght string) (*dagger.Directory, error) {
|
||||
ght, err := githubToken(ctx, ght)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error acquiring GitHub token: %w", err)
|
||||
}
|
||||
|
||||
// Clone the patches repository on 'main'
|
||||
dir, err := git.CloneWithGitHubToken(client, ght, repo, ref)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error cloning patches repository: %w", err)
|
||||
}
|
||||
|
||||
entries, err := dir.Entries(ctx, dagger.DirectoryEntriesOpts{
|
||||
Path: patchesPath,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error listing entries in repository: %w", err)
|
||||
}
|
||||
|
||||
if len(entries) == 0 {
|
||||
return nil, fmt.Errorf("no patches in the given path")
|
||||
}
|
||||
|
||||
container := client.Container().From(git.GitImage).
|
||||
WithEntrypoint([]string{}).
|
||||
WithMountedDirectory("/src", src).
|
||||
WithMountedDirectory("/patches", dir).
|
||||
WithWorkdir("/src").
|
||||
WithExec([]string{"git", "config", "--local", "user.name", "grafana"}).
|
||||
WithExec([]string{"git", "config", "--local", "user.email", "engineering@grafana.com"})
|
||||
|
||||
for _, v := range entries {
|
||||
if filepath.Ext(v) != ".patch" {
|
||||
continue
|
||||
}
|
||||
|
||||
container = container.WithExec([]string{"/bin/sh", "-c", fmt.Sprintf(`git am --3way --ignore-whitespace --ignore-space-change --committer-date-is-author-date %s > /dev/null 2>&1`, path.Join("/patches", patchesPath, v))})
|
||||
}
|
||||
|
||||
return container.Directory("/src"), nil
|
||||
}
|
||||
|
||||
func grafanaDirectory(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
o := GrafanaDirectoryOptsFromFlags(opts.CLIContext)
|
||||
|
||||
src, err := cloneOrMount(ctx, opts.Client, o.GrafanaDir, o.GrafanaRepo, o.GrafanaRef, o.GitHubToken)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
gitContainer := opts.Client.Container().From("alpine/git").
|
||||
WithWorkdir("/src").
|
||||
WithMountedDirectory("/src/.git", src.Directory(".git")).
|
||||
WithEntrypoint([]string{})
|
||||
|
||||
commitFile := gitContainer.
|
||||
WithExec([]string{"/bin/sh", "-c", "git rev-parse HEAD > .buildinfo.grafana-commit"}).
|
||||
File("/src/.buildinfo.grafana-commit")
|
||||
|
||||
branchFile := gitContainer.
|
||||
WithExec([]string{"/bin/sh", "-c", "git rev-parse --abbrev-ref HEAD > .buildinfo.grafana-branch"}).
|
||||
File("/src/.buildinfo.grafana-branch")
|
||||
|
||||
src = src.
|
||||
WithFile(".buildinfo.commit", commitFile).
|
||||
WithFile(".buildinfo.branch", branchFile)
|
||||
|
||||
if o.PatchesRepo != "" {
|
||||
withPatches, err := applyPatches(ctx, opts.Client, src, o.PatchesRepo, o.PatchesPath, o.PatchesRef, o.GitHubToken)
|
||||
if err != nil {
|
||||
opts.Log.Debug("patch application skipped", "error", err)
|
||||
} else {
|
||||
// Only replace src when there was no error.
|
||||
src = withPatches
|
||||
}
|
||||
}
|
||||
|
||||
nodeVersion, err := frontend.NodeVersion(opts.Client, src).Stdout(ctx)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("failed to get node version from source code: %w", err)
|
||||
}
|
||||
|
||||
yarnCache, err := opts.State.CacheVolume(ctx, YarnCacheDirectory)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
container := frontend.YarnInstall(opts.Client, src, nodeVersion, yarnCache, opts.Platform)
|
||||
|
||||
if _, err := containers.ExitError(ctx, container); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return container.Directory("/src"), nil
|
||||
}
|
||||
|
||||
func enterpriseDirectory(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
// Get the Grafana directory...
|
||||
o := GrafanaDirectoryOptsFromFlags(opts.CLIContext)
|
||||
|
||||
grafanaDir, err := grafanaDirectory(ctx, opts)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error initializing grafana directory: %w", err)
|
||||
}
|
||||
|
||||
clone, err := cloneOrMount(ctx, opts.Client, o.EnterpriseDir, o.EnterpriseRepo, o.EnterpriseRef, o.GitHubToken)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("error cloning or mounting Grafana Enterprise directory: %w", err)
|
||||
}
|
||||
|
||||
return InitializeEnterprise(opts.Client, grafanaDir.(*dagger.Directory), clone), nil
|
||||
}
|
||||
|
||||
var GrafanaDirectoryFlags = []cli.Flag{
|
||||
&cli.StringFlag{
|
||||
Name: "grafana-dir",
|
||||
Usage: "Local Grafana dir to use, instead of git clone",
|
||||
Required: false,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "enterprise-dir",
|
||||
Usage: "Local Grafana Enterprise dir to use, instead of git clone",
|
||||
Required: false,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "grafana-repo",
|
||||
Usage: "Grafana repo to clone, not valid if --grafana-dir is set",
|
||||
Required: false,
|
||||
Value: "https://github.com/grafana/grafana.git",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "enterprise-repo",
|
||||
Usage: "Grafana Enterprise repo to clone, not valid if --grafana-dir is set",
|
||||
Required: false,
|
||||
Value: "https://github.com/grafana/grafana-enterprise.git",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "grafana-ref",
|
||||
Usage: "Grafana ref to clone, not valid if --grafana-dir is set",
|
||||
Required: false,
|
||||
Value: "main",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "enterprise-ref",
|
||||
Usage: "Grafana Enterprise ref to clone, not valid if --grafana-dir is set",
|
||||
Required: false,
|
||||
Value: "main",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "github-token",
|
||||
Usage: "GitHub token to use for git cloning, by default will be pulled from GitHub",
|
||||
Required: false,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "patches-repo",
|
||||
Usage: "GitHub repository that contains git patches to apply to the Grafana source code. Must be an https git URL",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "patches-path",
|
||||
Usage: "Path to folder containing '.patch' files to apply",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "patches-ref",
|
||||
Usage: "Ref to checkout in the patches repository",
|
||||
Value: "main",
|
||||
},
|
||||
}
|
||||
|
||||
// GrafanaDirectory will provide the valueFunc that initializes and returns a *dagger.Directory that has Grafana in it.
|
||||
// Where possible, when cloning and no authentication options are provided, the valuefunc will try to use the configured github CLI for cloning.
|
||||
var GrafanaDirectory = pipeline.Argument{
|
||||
Name: "grafana-dir",
|
||||
Description: "The source tree of the Grafana repository",
|
||||
Flags: GrafanaDirectoryFlags,
|
||||
ValueFunc: grafanaDirectory,
|
||||
}
|
||||
|
||||
// EnterpriseDirectory will provide the valueFunc that initializes and returns a *dagger.Directory that has Grafana Enterprise initialized it.
|
||||
// Where possible, when cloning and no authentication options are provided, the valuefunc will try to use the configured github CLI for cloning.
|
||||
var EnterpriseDirectory = pipeline.Argument{
|
||||
Name: "enterprise-dir",
|
||||
Description: "The source tree of Grafana Enterprise",
|
||||
Flags: GrafanaDirectoryFlags,
|
||||
ValueFunc: enterpriseDirectory,
|
||||
}
|
||||
@@ -0,0 +1,70 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/cliutil"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var HGDirectoryFlags = []cli.Flag{
|
||||
&cli.StringFlag{
|
||||
Name: "hosted-grafana-dir",
|
||||
Usage: "Local clone of HG to use, instead of git cloning",
|
||||
Required: false,
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "hosted-grafana-repo",
|
||||
Usage: "https `.git` repository to use for hosted-grafana",
|
||||
Required: false,
|
||||
Value: "https://github.com/grafana/hosted-grafana.git",
|
||||
},
|
||||
&cli.StringFlag{
|
||||
Name: "hosted-grafana-ref",
|
||||
Usage: "git ref to checkout",
|
||||
Required: false,
|
||||
Value: "main",
|
||||
},
|
||||
}
|
||||
|
||||
// HGDirectory will provide the valueFunc that initializes and returns a *dagger.Directory that has a repository that has the Grafana Pro/Enterprise docker image.
|
||||
// Where possible, when cloning and no authentication options are provided, the valuefunc will try to use the configured github CLI for cloning.
|
||||
var HGDirectory = pipeline.Argument{
|
||||
Name: "hg-dir",
|
||||
Description: "The source tree of that has the Dockerfile for Grafana Pro/Enterprise",
|
||||
Flags: HGDirectoryFlags,
|
||||
ValueFunc: hgDirectory,
|
||||
}
|
||||
|
||||
type HGDirectoryOpts struct {
|
||||
GitHubToken string
|
||||
HGDir string
|
||||
HGRepo string
|
||||
HGRef string
|
||||
}
|
||||
|
||||
func HGDirectoryOptsFromFlags(c cliutil.CLIContext) *HGDirectoryOpts {
|
||||
return &HGDirectoryOpts{
|
||||
GitHubToken: c.String("github-token"),
|
||||
HGDir: c.String("hosted-grafana-dir"),
|
||||
HGRepo: c.String("hosted-grafana-repo"),
|
||||
HGRef: c.String("hosted-grafana-ref"),
|
||||
}
|
||||
}
|
||||
|
||||
func hgDirectory(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
o := HGDirectoryOptsFromFlags(opts.CLIContext)
|
||||
ght, err := githubToken(ctx, o.GitHubToken)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("could not get GitHub token: %w", err)
|
||||
}
|
||||
|
||||
src, err := cloneOrMount(ctx, opts.Client, o.HGDir, o.HGRepo, o.HGRef, ght)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return src, nil
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
package arguments
|
||||
|
||||
import "github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
|
||||
func Join(f ...[]pipeline.Argument) []pipeline.Argument {
|
||||
r := []pipeline.Argument{}
|
||||
for _, v := range f {
|
||||
r = append(r, v...)
|
||||
}
|
||||
|
||||
return r
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"strings"
|
||||
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/containers"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/stringutil"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var flagBuildID = &cli.StringFlag{
|
||||
Name: "build-id",
|
||||
Usage: "Build ID to use in package names",
|
||||
Value: "local",
|
||||
}
|
||||
|
||||
var BuildID = pipeline.Argument{
|
||||
Name: "build-id",
|
||||
Description: "The grafana backend binaries ('grafana', 'grafana-cli', 'grafana-server') in a directory",
|
||||
Flags: []cli.Flag{
|
||||
flagBuildID,
|
||||
},
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
v := opts.CLIContext.String("build-id")
|
||||
if v == "" {
|
||||
v = stringutil.RandomString(8)
|
||||
}
|
||||
|
||||
return v, nil
|
||||
},
|
||||
}
|
||||
|
||||
var flagVersion = &cli.StringFlag{
|
||||
Name: "version",
|
||||
Usage: "Explicit version number. If this is not set then one with will auto-detected based on the source repository",
|
||||
}
|
||||
|
||||
var Version = pipeline.Argument{
|
||||
Name: "version",
|
||||
Description: "The version string that is shown in the UI, in the CLI, and in package metadata",
|
||||
Flags: []cli.Flag{
|
||||
flagVersion,
|
||||
},
|
||||
Requires: []pipeline.Argument{
|
||||
GrafanaDirectory,
|
||||
},
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
v := opts.CLIContext.String("version")
|
||||
if v != "" {
|
||||
return v, nil
|
||||
}
|
||||
src, err := opts.State.Directory(ctx, GrafanaDirectory)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
buildID, err := opts.State.String(ctx, BuildID)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
version, err := containers.GetJSONValue(ctx, opts.Client, src, "package.json", "version")
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
return strings.ReplaceAll(version, "pre", buildID), nil
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
package arguments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
|
||||
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
|
||||
"github.com/urfave/cli/v2"
|
||||
)
|
||||
|
||||
var YarnCacheDirFlag = &cli.StringFlag{
|
||||
Name: "yarn-cache-dir",
|
||||
Aliases: []string{"yarn-cache"},
|
||||
Usage: "Path to the yarn cache directory to mount during 'yarn install' commands (if there is one)",
|
||||
EnvVars: []string{"YARN_CACHE_FOLDER", "YARN_CACHE_DIR"},
|
||||
Value: "",
|
||||
}
|
||||
|
||||
var YarnCacheDirectory = pipeline.Argument{
|
||||
Name: "yarn-cache-dir",
|
||||
Description: YarnCacheDirFlag.Usage,
|
||||
ArgumentType: pipeline.ArgumentTypeCacheVolume,
|
||||
Flags: []cli.Flag{
|
||||
YarnCacheDirFlag,
|
||||
},
|
||||
ValueFunc: func(ctx context.Context, opts *pipeline.ArgumentOpts) (any, error) {
|
||||
vol := opts.CLIContext.String(YarnCacheDirFlag.Name)
|
||||
|
||||
// Prepopulate the cache with what's defined in YARN_CACHE_FOLDER
|
||||
// or in the CLI
|
||||
if val, ok := os.LookupEnv("YARN_CACHE_FOLDER"); ok {
|
||||
vol = val
|
||||
}
|
||||
|
||||
cache := opts.Client.CacheVolume("yarn-cache-dir")
|
||||
if vol == "" {
|
||||
return cache, nil
|
||||
}
|
||||
|
||||
dir := opts.Client.Host().Directory(vol)
|
||||
_, err := opts.Client.Container().
|
||||
From("alpine").
|
||||
WithMountedCache("/cache", cache).
|
||||
WithMountedDirectory("/data", dir).
|
||||
WithExec([]string{"/bin/sh", "-c", "cp -r /data/* /cache || return 0"}).
|
||||
Sync(ctx)
|
||||
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return cache, nil
|
||||
},
|
||||
}
|
||||
Reference in New Issue
Block a user