Files
grafana/pkg/build/daggerbuild/artifacts/package_docker.go
T
Jev Forsberg dc6bd2a20a CI Migration: Backport pkg/build (daggerbuild) and GHAs to release-12.0.3 (#107997)
* CI: move grafana-build into pkg/build (#105640)

* move grafana-build into pkg/build

* Update go.mod/go.sum files after cherry-pick build

- Updates from workspace sync after cherry-picking daggerbuild package
- Resolves dependency conflicts from pkg/build integration
- Required for successful build after CI migration backport

* CI Migration: Replace .github with main branch version

- Complete replacement of .github directory from main branch
- Includes updated workflows, actions, and configuration files
- Ensures all CI improvements from main are included
- Next step: fix branch-specific issues in separate commits

* Fix branch triggers: Update workflows to target release branches

- Update 8 workflows to trigger on release-* instead of main
- Ensures workflows run on pull requests to release branches
- Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator

* Fix branch triggers: Update workflows to target both main and release branches

- CORRECTED: Include both main and release-* branch patterns
- Ensures workflows run on both main branch (original functionality) and release branches (needed for backport)
- Previous commit incorrectly removed main branch support
- Workflows updated: documentation-ci, pr-codeql-analysis-*, pr-dependabot-update-go-workspace, pr-go-workspace-check, pr-k8s-codegen-check, verify-kinds, codeowners-validator

* CI Migration: Update .gitignore to allow OSS wire file to be committed

- Remove **/wire_gen.go (allow pkg/server/wire_gen.go to be committed)
- Add /pkg/server/enterprise_wire_gen.go (keep enterprise wire file ignored)
- This enables the new committed wire files architecture for CI

* CI Migration: Add enhanced wire tool with automatic build constraints

- Backport enhanced wire tool from main branch with -gen_tags flag
- This enables automatic regeneration with proper constraints to prevent conflicts
- Wire file contains all required functions: Initialize, InitializeDocumentBuilders, etc.
- Solves the missing wire functions issue causing CI failures
- Enhanced tool ensures build constraints persist on regeneration

* CI Migration: Update Makefile to use enhanced wire tool with build constraints

- Update gen-go target to use -gen_tags flag for automatic build constraints
- Ensures future engineers get proper build constraints when running 'make gen-go'
- Matches main branch approach for consistent wire file generation
- Removes dependency on WIRE_TAGS variable and GO_RACE_FLAG for simplicity

* Add gen-enterprise-go target to Makefile for release-12.0.3

- Backports gen-enterprise-go target from main branch
- Enables enterprise wire file generation in release branches
- Part of CI migration backport wire infrastructure fixes
- Resolves CI issues where enterprise wire functions were missing

* Fix wire tool golden file format for Drone CI compatibility

- Update all 43 golden test files to match main branch format
- Change go:generate command from github.com/google/wire/cmd/wire to ./pkg/build/wire/cmd/wire/main.go
- Remove legacy '// +build !wireinject' constraints
- Fixes Drone CI test failures that were blocking OSS PR merge
- All wire tests now pass successfully

This aligns the backport branch with the golden file format updates
that were made in main branch when the wire tool was enhanced.

* Update swagger specs to include enterprise APIs

- Regenerated api-enterprise-spec.json with enterprise API definitions
- Updated api-merged.json with merged OSS and enterprise specs
- Regenerated openapi3.json with complete API specifications
- Fixes enterprise CI swagger generation validation failures
- Enterprise APIs now properly detected and documented

* Fix OSS integration tests missing enterprise build tags in Drone CI

- Add -tags=enterprise to all integration test steps in lib.star
- Fixes test-backend-integration, postgres-integration-tests, and mysql-integration-tests
- Regenerate .drone.yml with proper enterprise build tags
- Resolves 'server could not find the requested resource' errors for enterprise APIs
- Ensures enterprise APIs (querylibrary, reporting, banners, scim) are registered during tests
- Aligns OSS Drone CI with GitHub Actions behavior and enterprise repository

* Fix CODEOWNERS for release-12.0.3 compatibility

- Remove 21 entries that reference files/directories not present in release branch
- Resolves File Exist Checker failures in codeowners-validator workflow
- Maintains team ownership assignments from main branch for existing files
- Lines removed: .air.toml, apps/secret/, apps/iam/, e2e-playwright/, packages/grafana-alerting/, etc.

* baldm0mma/ update releasefinder

* Add i18n-extract script for release branch compatibility

- Add 'i18n-extract': 'make i18n-extract' to package.json scripts
- Resolves i18n-verify workflow failure that expects this script
- The make target already exists and handles OSS + enterprise i18n extraction
- Maintains i18n verification functionality from main branch workflows
- Simple safe addition that calls existing make infrastructure

* Phase 4: Fix GitHub Actions workflow branch triggers

- Add release-*.*.* pattern to 8 workflows missing it
- Fix duplicate release branch patterns in 3 workflows
- Ensure consistent branch trigger format across all workflows
- Critical workflows now properly trigger on release branches

Fixes workflows: actionlint, backend-code-checks, go-lint, reject-gh-secrets,
run-schema-v2-e2e, shellcheck, swagger-gen, trivy-scan

* Fix: Backport E2E runner infrastructure for Enterprise CI

- Add e2e/main.go and e2e/internal/ directory from main branch
- Add urfave/cli/v3 dependency required by E2E runner
- Fixes Enterprise CI failure: 'Build E2E test runner' and 'Build & package Grafana for e2e tests'
- Root cause: E2E runner infrastructure was added to main after release-12.0.3 branch creation
- Solution: Backport missing E2E runner files to enable Enterprise CI completion

Resolves: no Go files in /opt/actions-runner/_work/grafana-enterprise/grafana-enterprise/grafana/e2e

* Fix: Update go.mod dependency classification for urfave/cli/v3

- Change urfave/cli/v3 from indirect to direct dependency
- Fixes Go Workspace Check failure in CI
- Required after adding E2E runner infrastructure that directly imports urfave/cli/v3
- Resolves: make update-workspace corrects dependency classification

* Fix: Add team ownership for urfave/cli/v3 dependency

- Assign @grafana/grafana-backend-group as owner for github.com/urfave/cli/v3@v3.3.8
- Follows existing pattern: urfave/cli v1 and v2 also owned by grafana-backend-group
- Resolves Backend Code Checks / Validate Backend Configs CI failure
- Required for E2E runner infrastructure dependency ownership compliance

Fixes: modowners check requiring team assignment for newly added dependencies

* Fix: Revert experimental E2E playwright infrastructure to stable version

- Revert pr-e2e-tests.yml to stable Cypress-based E2E testing
- Remove experimental storybook-verification-playwright.yml workflow
- Revert run-dashboard-search-e2e.yml and release-pr.yml to use e2e/test-plugins/
- Keep stable E2E runner infrastructure (e2e/main.go + e2e/internal/)
- Remove experimental playwright features per team recommendation

Team feedback: Playwright tests are experimental and shouldn't be backported to stable release branches

* Fix: Complete cleanup of experimental playwright dependencies

- Revert package.json to stable version (remove e2e-playwright scripts and path references)
- Revert playwright.config.ts to stable plugin-e2e configuration
- Remove all experimental playwright infrastructure dependencies
- Ensure clean stable E2E testing environment

All experimental features removed per team recommendation for stable release branches

* Fix: Restore working package.json configuration

- Revert package.json to version 12.0.3 (working release branch version)
- Fix workspace dependency resolution issues caused by incorrect revert to main branch version
- FE tests, betterer, and linting should now work correctly

Issue was caused by reverting package.json to main branch (12.1.0-pre) instead of keeping
the working release branch configuration (12.0.3)

* CI: mirror some CI dependencies (#106148)

* mirror some CI dependencies
* remove -v from go build

* Fix: Backport missing e2e/run-suite script for daggerbuild E2E system

The daggerbuild E2E system (used by OSS workflows) expects ./e2e/run-suite
to exist, but we only backported the new E2E runner infrastructure.

This script is needed for:
- OSS workflow: 'go run ./pkg/build/e2e --suite=dashboards-suite'
- Legacy Cypress configuration with video support
- Integration between daggerbuild and existing Cypress test suites

Resolves E2E test failures in OSS workflows where videos directory
cannot be found because run-suite script was missing.

* Fix: Resolve daggerbuild E2E path doubling issue

- Extract just suite name from full path using filepath.Base()
- Prevents doubled paths like './e2e/e2e/dashboards-suite/videos'
- Resolves 'no spec files found' and 'no such file or directory' errors
- GitHub Actions passes full paths like 'e2e/dashboards-suite' but run-suite script expects just 'dashboards-suite'

* Infrastructure: Wholesale copy pkg/build/ from main

COMPLETE DAGGERBUILD SYSTEM UPDATE:
- Modern E2E system with CLI framework and --flags support
- New accessibility testing system (a11y/)
- New Playwright E2E testing system (e2e-playwright/)
- External infrastructure improvements (musl.cc → dl.grafana.com/ci)
- Updated daggerbuild core components with latest fixes
- Updated Go dependencies and wire modules

ARCHITECTURAL COMPATIBILITY VERIFIED:
- Modern pkg/build calls: ./e2e-runner cypress --start-grafana=false --cypress-video
- Our e2e runner supports: All required parameters including --suite, --env flags
- GitHub Actions workflows: Pass compatible arguments
- Binary builds: Successfully creates working e2e-runner

RESOLVES ISSUES:
- Fixes --flags parameter compatibility with GitHub Actions
- Includes all follow-up infrastructure improvements
- Provides complete, tested system with modern CLI framework
- Eliminates external dependency failures (musl.cc timeouts)
- Removes need for path doubling workarounds (modern system handles full paths correctly)

Replaces incremental cherry-picking with complete, tested system from main.

* Dependencies: Update Go modules after pkg/build wholesale copy

DEPENDENCY UPDATES:
- Updated go.work.sum with new dependencies from modern pkg/build system
- Updated all workspace module dependencies (go.mod/go.sum files)
- Removed pkg/build/cmd/enterprise.go (enterprise-only file, gets copied during enterprise development)

ENTERPRISE INTEGRATION FIX:
- enterprise.go file doesn't belong in OSS repository
- Gets copied from grafana-enterprise during development mode
- Main branch doesn't have this file, explaining module compatibility

All Go modules now properly resolved and compatible with modern pkg/build architecture.

* Dependencies: Update workspace Go module checksums after pkg/build wholesale copy

- Synchronizes all go.sum files across workspace modules
- Adds missing .mod.h1 entries that were required after infrastructure update
- Resolves Go Workspace Check CI failures
- Updates 22 modules: .citools/, apps/, pkg/ subdirectories plus go.work.sum

Addresses CI error: 'Please run make update-workspace and commit the changes'

* Infrastructure: Revert to pre-Playwright E2E system for release branch compatibility

Strategic combination of modern daggerbuild with stable E2E infrastructure:

**Modern Daggerbuild (from main):**
- Latest CLI framework (github.com/urfave/cli/v3)
- External infrastructure fixes (musl.cc → dl.grafana.com/ci)
- Complete pkg/build/ system with all enhancements
- Updated dependencies and architecture improvements

**Stable E2E Infrastructure (pre-Playwright):**
- pkg/build/e2e/service.go: Reverted to version before commit b6580ccb10
- pkg/build/a11y/: Reverted to stable version compatible with existing infrastructure
- Removed experimental pkg/build/e2e-playwright/ system
- Removed experimental e2e-playwright/ test infrastructure

**Why This Approach:**
- GitHub Actions workflows call 'go run ./pkg/build/e2e' (modern system)
- Modern system now expects only standard e2e/ directory (compatible)
- Release branches have standard e2e/ infrastructure (dashboards-suite, various-suite, etc.)
- Eliminates dependency on experimental Playwright infrastructure

**Key Dependencies Met:**
- e2e/pa11yci.conf.js: Added for a11y test compatibility
- /src/e2e directory: Standard directory structure (no e2e-playwright needed)
- Existing test plugins: Uses stable e2e/test-plugins/

**Result:**
Best of both worlds - modern, reliable daggerbuild infrastructure with
proven E2E testing that works across all release branches without
requiring experimental dependencies.

* Fix: Remove unused embed.go file causing golangci-lint failure

- Removes pkg/build/daggerbuild/msi/embed.go with unused embed.FS variable
- MSI build process uses directory mounting instead of embedded filesystem
- Resolves 'var resources is unused (unused)' linting error

* Fix: Update Node.js version to match working E2E configuration

- Changes from v22.11.0 to v22.16.0 to match commit 17952d45e4
- Working commit had all E2E tests passing with same daggerbuild infrastructure
- Node.js version differences can affect frontend builds and UI rendering
- Ensures consistent environment between working reference and current branch

* Fix: Use conditional container base for E2E compatibility

- Alpine for OSS tests (better old arch dashboardScene=false compatibility)
- Ubuntu for Enterprise tests (when image renderer needed)
- Restores original release-12.0.3 Alpine behavior for OSS
- Should resolve old arch E2E test failures

* Fix: Go fmt formatting for conditional container logic

* Revert: Remove unnecessary Alpine/Ubuntu conditional logic

- Real issue was wrong environment variable name (dashboardScene vs DISABLE_SCENES)
- Keep Ubuntu base (original main branch behavior) for simplicity
- Container base was not the root cause of old arch failures

* Experiment: Copy scripts/ from pre-Playwright commit (ea0ddb3fc9)

- Copied scripts directory from commit ea0ddb3fc9 (just before Playwright migration)
- This overwrites some CI migration fixes but testing if it's actually needed
- Can revert back to b9b4602dbd if CI breaks

Test commit to see real impact on GitHub Actions workflows.

* trigger ci 01

* Fix: Copy .drone.star from main to match scripts structure

- Resolves Starlark evaluation error for missing integration_test_pipelines
- Aligns .drone.star with main branch scripts structure
- Same fix applied to both OSS and Enterprise repositories

* make drone

* Fix ARM/v7 Docker build failures by using dagger run for cross-compilation

- Change artifacts_cmd() in scripts/drone/steps/rgm.star to use 'dagger run go run' instead of 'go run'
- Add --max-execution-steps 100000 to make drone target in Makefile to properly regenerate .drone.yml
- This fixes CGO cross-compilation issues with SQLite for ARM/v7 Docker builds
- Matches the approach already used successfully in Enterprise builds

* Fix Docker base image references in rgm build steps

- Change from symbolic names (alpine-base, ubuntu-base) to actual Docker images
- alpine-base → alpine:3.21.3
- ubuntu-base → ubuntu:22.04
- Add ubuntu and alpine parameters to rgm_artifacts_step and rgm_build_docker_step functions
- Fixes Docker pull errors: 'repository does not exist or may require authorization'
- Aligns OSS configuration with Enterprise approach

---------

Co-authored-by: Kevin Minehart <5140827+kminehart@users.noreply.github.com>
2025-07-18 09:14:14 -06:00

266 lines
7.6 KiB
Go

package artifacts
import (
"context"
"fmt"
"log/slog"
"strings"
"dagger.io/dagger"
"github.com/grafana/grafana/pkg/build/daggerbuild/arguments"
"github.com/grafana/grafana/pkg/build/daggerbuild/backend"
"github.com/grafana/grafana/pkg/build/daggerbuild/docker"
"github.com/grafana/grafana/pkg/build/daggerbuild/flags"
"github.com/grafana/grafana/pkg/build/daggerbuild/packages"
"github.com/grafana/grafana/pkg/build/daggerbuild/pipeline"
)
var (
DockerArguments = arguments.Join(
TargzArguments,
[]pipeline.Argument{
arguments.DockerRegistry,
arguments.DockerOrg,
arguments.AlpineImage,
arguments.UbuntuImage,
arguments.TagFormat,
arguments.UbuntuTagFormat,
arguments.BoringTagFormat,
},
)
DockerFlags = flags.JoinFlags(
TargzFlags,
flags.DockerFlags,
)
)
var DockerInitializer = Initializer{
InitializerFunc: NewDockerFromString,
Arguments: DockerArguments,
}
// PacakgeDocker uses a built tar.gz package to create a docker image from the Dockerfile in the tar.gz
type Docker struct {
Name packages.Name
Version string
BuildID string
Distro backend.Distribution
Enterprise bool
Ubuntu bool
Registry string
Repositories []string
Org string
BaseImage string
TagFormat string
Tarball *pipeline.Artifact
// Src is the Grafana source code for running e2e tests when validating.
// The grafana source should not be used for anything else when building a docker image. All files in the Docker image, including the Dockerfile, should be
// from the tar.gz file.
Src *dagger.Directory
YarnCache *dagger.CacheVolume
}
func (d *Docker) Dependencies(ctx context.Context) ([]*pipeline.Artifact, error) {
return []*pipeline.Artifact{
d.Tarball,
}, nil
}
func (d *Docker) Builder(ctx context.Context, opts *pipeline.ArtifactContainerOpts) (*dagger.Container, error) {
targz, err := opts.Store.File(ctx, d.Tarball)
if err != nil {
return nil, err
}
return docker.Builder(opts.Client, opts.Client.Host().UnixSocket("/var/run/docker.sock"), targz), nil
}
func (d *Docker) BuildFile(ctx context.Context, builder *dagger.Container, opts *pipeline.ArtifactContainerOpts) (*dagger.File, error) {
// Unlike most other things we push to, docker image tags do not support all characters.
// Specifically, the `+` character used in the `buildmetadata` section of semver.
version := strings.ReplaceAll(d.Version, "+", "-")
tags, err := docker.Tags(d.Org, d.Registry, d.Repositories, d.TagFormat, packages.NameOpts{
Name: d.Name,
Version: version,
BuildID: d.BuildID,
Distro: d.Distro,
})
if err != nil {
return nil, err
}
buildOpts := &docker.BuildOpts{
// Tags are provided as the '-t' argument, and can include the registry domain as well as the repository.
// Docker build supports building the same image with multiple tags.
// You might want to also include a 'latest' version of the tag.
Tags: tags,
Platform: backend.Platform(d.Distro),
BuildArgs: []string{
"GRAFANA_TGZ=grafana.tar.gz",
"GO_SRC=tgz-builder",
"JS_SRC=tgz-builder",
fmt.Sprintf("BASE_IMAGE=%s", d.BaseImage),
},
}
b := docker.Build(opts.Client, builder, buildOpts)
return docker.Save(b, buildOpts), nil
}
func (d *Docker) BuildDir(ctx context.Context, builder *dagger.Container, opts *pipeline.ArtifactContainerOpts) (*dagger.Directory, error) {
panic("This artifact does not produce directories")
}
func (d *Docker) Publisher(ctx context.Context, opts *pipeline.ArtifactContainerOpts) (*dagger.Container, error) {
socket := opts.Client.Host().UnixSocket("/var/run/docker.sock")
return opts.Client.Container().From("docker").WithUnixSocket("/var/run/docker.sock", socket), nil
}
func (d *Docker) PublishFile(ctx context.Context, opts *pipeline.ArtifactPublishFileOpts) error {
panic("not implemented")
}
func (d *Docker) PublishDir(ctx context.Context, opts *pipeline.ArtifactPublishDirOpts) error {
panic("This artifact does not produce directories")
}
// Filename should return a deterministic file or folder name that this build will produce.
// This filename is used as a map key for caching, so implementers need to ensure that arguments or flags that affect the output
// also affect the filename to ensure that there are no collisions.
// For example, the backend for `linux/amd64` and `linux/arm64` should not both produce a `bin` folder, they should produce a
// `bin/linux-amd64` folder and a `bin/linux-arm64` folder. Callers can mount this as `bin` or whatever if they want.
func (d *Docker) Filename(ctx context.Context) (string, error) {
ext := "docker.tar.gz"
if d.Ubuntu {
ext = "ubuntu.docker.tar.gz"
}
return packages.FileName(d.Name, d.Version, d.BuildID, d.Distro, ext)
}
func (d *Docker) VerifyFile(ctx context.Context, client *dagger.Client, file *dagger.File) error {
// Currently verifying riscv64 is unsupported (because alpine and ubuntu don't have riscv64 images yet)
if _, arch := backend.OSAndArch(d.Distro); arch == "riscv64" {
return nil
}
return docker.Verify(ctx, client, file, d.Src, d.YarnCache, d.Distro)
}
func (d *Docker) VerifyDirectory(ctx context.Context, client *dagger.Client, dir *dagger.Directory) error {
panic("not implemented") // TODO: Implement
}
func NewDockerFromString(ctx context.Context, log *slog.Logger, artifact string, state pipeline.StateHandler) (*pipeline.Artifact, error) {
options, err := pipeline.ParseFlags(artifact, DockerFlags)
if err != nil {
return nil, err
}
p, err := GetPackageDetails(ctx, options, state)
if err != nil {
return nil, err
}
tarball, err := NewTarballFromString(ctx, log, artifact, state)
if err != nil {
return nil, err
}
ubuntu, err := options.Bool(flags.Ubuntu)
if err != nil {
return nil, err
}
// Ubuntu Version to use as the base for the Grafana docker image (if this is a ubuntu artifact)
// This shouldn't fail if it's not set by the user, instead it'll default to 22.04 or something.
ubuntuImage, err := state.String(ctx, arguments.UbuntuImage)
if err != nil {
return nil, err
}
// Same for Alpine
alpineImage, err := state.String(ctx, arguments.AlpineImage)
if err != nil {
return nil, err
}
registry, err := state.String(ctx, arguments.DockerRegistry)
if err != nil {
return nil, err
}
org, err := state.String(ctx, arguments.DockerOrg)
if err != nil {
return nil, err
}
repos, err := options.StringSlice(flags.DockerRepositories)
if err != nil {
return nil, err
}
format, err := state.String(ctx, arguments.TagFormat)
if err != nil {
return nil, err
}
ubuntuFormat, err := state.String(ctx, arguments.UbuntuTagFormat)
if err != nil {
return nil, err
}
boringFormat, err := state.String(ctx, arguments.BoringTagFormat)
if err != nil {
return nil, err
}
base := alpineImage
if ubuntu {
format = ubuntuFormat
base = ubuntuImage
}
if p.Name == packages.PackageEnterpriseBoring {
format = boringFormat
}
src, err := state.Directory(ctx, arguments.GrafanaDirectory)
if err != nil {
return nil, err
}
yarnCache, err := state.CacheVolume(ctx, arguments.YarnCacheDirectory)
if err != nil {
return nil, err
}
log.Info("initializing Docker artifact", "Org", org, "registry", registry, "repos", repos, "tag", format)
return pipeline.ArtifactWithLogging(ctx, log, &pipeline.Artifact{
ArtifactString: artifact,
Handler: &Docker{
Name: p.Name,
Version: p.Version,
BuildID: p.BuildID,
Distro: p.Distribution,
Enterprise: p.Enterprise,
Tarball: tarball,
Ubuntu: ubuntu,
BaseImage: base,
Registry: registry,
Org: org,
Repositories: repos,
TagFormat: format,
Src: src,
YarnCache: yarnCache,
},
Type: pipeline.ArtifactTypeFile,
Flags: DockerFlags,
})
}