7e45a300b9
* Deprecate Legacy Storage Migration in Backend * Change the messaging around legacy storage * Disable cards to connect * Commit import changes * Block repository creation if resources are in legacy storage * Update error message * Prettify * chore: uncomment unified migration * chore: adapt and fix tests * Remove legacy storage migration from frontend * Refactor provisioning job options by removing legacy storage and history fields - Removed the `History` field from `MigrateJobOptions` and related references in the codebase. - Eliminated the `LegacyStorage` field from `RepositoryViewList` and its associated comments. - Updated tests and generated OpenAPI schema to reflect these changes. - Simplified the `MigrationWorker` by removing dependencies on legacy storage checks. * Refactor OpenAPI schema and tests to remove deprecated fields - Removed the `history` field from `MigrateJobOptions` and updated the OpenAPI schema accordingly. - Eliminated the `legacyStorage` field from `RepositoryViewList` and its associated comments in the schema. - Updated integration tests to reflect the removal of these fields. * Fix typescript errors * Refactor provisioning code to remove legacy storage dependencies - Eliminated references to `dualwrite.Service` and related legacy storage checks across multiple files. - Updated `APIBuilder`, `RepositoryController`, and `SyncWorker` to streamline resource handling without legacy storage considerations. - Adjusted tests to reflect the removal of legacy storage mocks and dependencies, ensuring cleaner and more maintainable code. * Fix unit tests * Remove more references to legacy * Enhance provisioning wizard with migration options - Added a checkbox for migrating existing resources in the BootstrapStep component. - Updated the form context to track the new migration option. - Adjusted the SynchronizeStep and useCreateSyncJob hook to incorporate the migration logic. - Enhanced localization with new descriptions and labels for migration features. * Remove unused variable and dualwrite reference in provisioning code - Eliminated an unused variable declaration in `provisioning_manifest.go`. - Removed the `nil` reference for dualwrite in `repo_operator.go`, aligning with the standalone operator's assumption of unified storage. * Update go.mod and go.sum to include new dependencies - Added `github.com/grafana/grafana-app-sdk` version `0.48.5` and several indirect dependencies including `github.com/getkin/kin-openapi`, `github.com/hashicorp/errwrap`, and others. - Updated `go.sum` to reflect the new dependencies and their respective versions. * Refactor provisioning components for improved readability - Simplified the import statement in HomePage.tsx by removing unnecessary line breaks. - Consolidated props in the SynchronizeStep component for cleaner code. - Enhanced the layout of the ProvisioningWizard component by streamlining the rendering of the SynchronizeStep. * Deprecate MigrationWorker and clean up related comments - Removed the deprecated MigrationWorker implementation and its associated comments from the provisioning code. - This change reflects the ongoing effort to eliminate legacy components and improve code maintainability. * Fix linting issues * Add explicit comment * Update useResourceStats hook in BootstrapStep component to accept selected target - Modified the BootstrapStep component to pass the selected target to the useResourceStats hook. - Updated related tests to reflect the change in expected arguments for the useResourceStats hook. * fix(provisioning): Update migrate tests to match export-then-sync behavior for all repository types Updates test expectations for folder-type repositories to match the implementation changes where both folder and instance repository types now run export followed by sync. Only the namespace cleaner is skipped for folder-type repositories. Changes: - Update "should run export and sync for folder-type repositories" test to include export mocks - Update "should fail when sync job fails for folder-type repositories" test to include export mocks - Rename test to clarify that both export and sync run for folder types - Add proper mock expectations for SetMessage, StrictMaxErrors, Process, and ResetResults All migrate package tests now pass. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Update provisioning wizard text and improve resource counting display - Enhanced descriptions for migrating existing resources to clarify that unmanaged resources will also be included. - Refactored BootstrapStepResourceCounting component to simplify the rendering logic and ensure both external storage and unmanaged resources are displayed correctly. - Updated alert messages in SynchronizeStep to reflect accurate information regarding resource management during migration. - Adjusted localization strings for consistency with the new descriptions. * Update provisioning wizard alert messages for clarity and accuracy - Revised alert points to indicate that resources can still be modified during migration, with a note on potential export issues. - Clarified that resources will be marked as managed post-provisioning and that dashboards remain accessible throughout the process. * Fix issue with trigger wrong type of job * Fix export failure when folder already exists in repository When exporting resources to a repository, if a folder already exists, the Read() method would fail with "path component is empty" error. This occurred because: 1. Folders are identified by trailing slash (e.g., "Legacy Folder/") 2. The Read() method passes this path directly to GetTreeByPath() 3. GetTreeByPath() splits the path by "/" creating empty components 4. This causes the "path component is empty" error The fix strips the trailing slash before calling GetTreeByPath() to avoid empty path components, while still using the trailing slash convention to identify directories. The Create() method already handles this correctly by appending ".keep" to directory paths, which is why the first export succeeded but subsequent exports failed. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Fix folder tree not updated when folder already exists in repository When exporting resources and a folder already exists in the repository, the folder was not being added to the FolderManager's tree. This caused subsequent dashboard exports to fail with "folder NOT found in tree". The fix adds the folder to fm.tree even when it already exists in the repository, ensuring all folders are available for resource lookups. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Revert "Merge remote-tracking branch 'origin/uncomment-unified-migration-code' into cleanup/deprecate-legacy-storage-migration-in-provisioning" This reverts commit6440fae342, reversing changes made toec39fb04f2. * fix: handle empty folder titles in path construction - Skip folders with empty titles in dirPath to avoid empty path components - Skip folders with empty paths before checking if they exist in repository - Fix unit tests to properly check useResourceStats hook calls with type annotations * Update workspace * Fix BootstrapStep tests after reverting unified migration merge Updated test expectations to match the current component behavior where resource counts are displayed for both instance and folder sync options. - Changed 'Empty' count expectation from 3 to 4 (2 cards × 2 counts each) - Changed '7 resources' test to use findAllByText instead of findByText since the count appears in multiple cards 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> * Remove bubbletee deps * Fix workspace * provisioning: update error message to reference enableMigration config Update the error message when provisioning cannot be used due to incompatible data format to instruct users to enable data migration for folders and dashboards using the enableMigration configuration introduced in PR #114857. Also update the test helper to include EnableMigration: true for both dashboards and folders to match the new configuration pattern. * provisioning: add comment explaining Mode5 and EnableMigration requirement Add a comment in the integration test helper explaining that Provisioning requires Mode5 (unified storage) and EnableMigration (data migration) as it expects resources to be fully migrated to unified storage. * Remove migrate resources checkbox from folder type provisioning wizard - Remove checkbox UI for migrating existing resources in folder type - Remove migrateExistingResources from migration logic - Simplify migration to only use requiresMigration flag - Remove unused translation keys - Update i18n strings * Fix linting * Remove unnecessary React Fragment wrapper in BootstrapStep * Address comments --------- Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com> Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
345 lines
11 KiB
Go
345 lines
11 KiB
Go
package resources
|
|
|
|
import (
|
|
"bytes"
|
|
"context"
|
|
"errors"
|
|
"fmt"
|
|
"slices"
|
|
"sync"
|
|
|
|
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
|
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
|
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
|
"k8s.io/apimachinery/pkg/runtime/schema"
|
|
"k8s.io/apimachinery/pkg/util/validation/field"
|
|
|
|
"github.com/grafana/grafana/apps/provisioning/pkg/repository"
|
|
"github.com/grafana/grafana/apps/provisioning/pkg/safepath"
|
|
"github.com/grafana/grafana/pkg/apimachinery/utils"
|
|
"github.com/grafana/grafana/pkg/infra/slugify"
|
|
"github.com/grafana/grafana/pkg/infra/tracing"
|
|
)
|
|
|
|
var (
|
|
ErrAlreadyInRepository = errors.New("already in repository")
|
|
ErrDuplicateName = errors.New("duplicate name in repository")
|
|
ErrMissingName = field.Required(field.NewPath("name", "metadata", "name"), "missing name in resource")
|
|
)
|
|
|
|
// NewResourceOwnershipConflictError creates a BadRequest error for when a resource
|
|
// is owned by a different repository or manager and cannot be modified
|
|
func NewResourceOwnershipConflictError(resourceName string, currentManager utils.ManagerProperties, requestingManager utils.ManagerProperties) error {
|
|
message := fmt.Sprintf("resource '%s' is managed by %s '%s' and cannot be modified by %s '%s'",
|
|
resourceName,
|
|
currentManager.Kind,
|
|
currentManager.Identity,
|
|
requestingManager.Kind,
|
|
requestingManager.Identity)
|
|
|
|
return apierrors.NewBadRequest(message)
|
|
}
|
|
|
|
type WriteOptions struct {
|
|
Path string
|
|
Ref string
|
|
}
|
|
|
|
type resourceID struct {
|
|
Name string
|
|
Resource string
|
|
Group string
|
|
}
|
|
|
|
type ResourcesManager struct {
|
|
repo repository.ReaderWriter
|
|
folders *FolderManager
|
|
parser Parser
|
|
clients ResourceClients
|
|
resourcesLookup map[resourceID]string // the path with this k8s name
|
|
mu sync.RWMutex
|
|
}
|
|
|
|
func NewResourcesManager(repo repository.ReaderWriter, folders *FolderManager, parser Parser, clients ResourceClients) *ResourcesManager {
|
|
return &ResourcesManager{
|
|
repo: repo,
|
|
folders: folders,
|
|
parser: parser,
|
|
clients: clients,
|
|
resourcesLookup: map[resourceID]string{},
|
|
}
|
|
}
|
|
|
|
// findResource checks if a resource exists in the lookup map (read operation)
|
|
func (r *ResourcesManager) findResource(id resourceID) (string, bool) {
|
|
r.mu.RLock()
|
|
defer r.mu.RUnlock()
|
|
path, found := r.resourcesLookup[id]
|
|
return path, found
|
|
}
|
|
|
|
func (r *ResourcesManager) addResource(id resourceID, path string) {
|
|
r.mu.Lock()
|
|
defer r.mu.Unlock()
|
|
|
|
if _, found := r.resourcesLookup[id]; found {
|
|
return
|
|
}
|
|
|
|
r.resourcesLookup[id] = path
|
|
}
|
|
|
|
// CheckResourceOwnership validates that the requesting manager can modify the existing resource
|
|
// Returns an error if the existing resource is owned by a different manager that doesn't allow edits
|
|
// If existingResource is nil, no ownership conflict exists (new resource)
|
|
// This is a package-level function that can be used without a ResourcesManager instance
|
|
func CheckResourceOwnership(existingResource *unstructured.Unstructured, resourceName string, requestingManager utils.ManagerProperties) error {
|
|
if existingResource == nil {
|
|
// Resource doesn't exist, so no ownership conflict
|
|
return nil
|
|
}
|
|
|
|
// Check if the existing resource has manager properties
|
|
existingMeta, err := utils.MetaAccessor(existingResource)
|
|
if err != nil {
|
|
// If we can't get metadata, allow the operation
|
|
return nil
|
|
}
|
|
|
|
currentManager, hasManager := existingMeta.GetManagerProperties()
|
|
if !hasManager {
|
|
// No manager information, so no ownership conflict
|
|
return nil
|
|
}
|
|
|
|
// Check if this is the same manager
|
|
if currentManager.Kind == requestingManager.Kind && currentManager.Identity == requestingManager.Identity {
|
|
// Same manager, no conflict
|
|
return nil
|
|
}
|
|
|
|
// Check if the current manager allows edits
|
|
if currentManager.AllowsEdits {
|
|
// Manager allows edits from others, no conflict
|
|
return nil
|
|
}
|
|
|
|
// Different manager and edits not allowed - return ownership conflict error
|
|
return NewResourceOwnershipConflictError(resourceName, currentManager, requestingManager)
|
|
}
|
|
|
|
// CreateResource writes an object to the repository
|
|
func (r *ResourcesManager) WriteResourceFileFromObject(ctx context.Context, obj *unstructured.Unstructured, options WriteOptions) (string, error) {
|
|
if err := ctx.Err(); err != nil {
|
|
return "", fmt.Errorf("context error: %w", err)
|
|
}
|
|
|
|
meta, err := utils.MetaAccessor(obj)
|
|
if err != nil {
|
|
return "", fmt.Errorf("extract meta accessor: %w", err)
|
|
}
|
|
|
|
// Message from annotations
|
|
commitMessage := meta.GetMessage()
|
|
if commitMessage == "" {
|
|
g := meta.GetGeneration()
|
|
if g > 0 {
|
|
commitMessage = fmt.Sprintf("Generation: %d", g)
|
|
} else {
|
|
commitMessage = "exported from grafana"
|
|
}
|
|
}
|
|
|
|
name := meta.GetName()
|
|
if name == "" {
|
|
return "", ErrMissingName
|
|
}
|
|
|
|
manager, _ := meta.GetManagerProperties()
|
|
// TODO: how should we handle this?
|
|
if manager.Identity == r.repo.Config().GetName() {
|
|
// If it's already in the repository, we don't need to write it
|
|
return "", ErrAlreadyInRepository
|
|
}
|
|
|
|
title := meta.FindTitle("")
|
|
if title == "" {
|
|
title = name
|
|
}
|
|
|
|
folder := meta.GetFolder()
|
|
// Get the absolute path of the folder
|
|
rootFolder := RootFolder(r.repo.Config())
|
|
|
|
// If no folder is specified in the file, set it to the root to ensure everything is written under it
|
|
var fid Folder
|
|
if folder == "" {
|
|
fid = Folder{ID: rootFolder}
|
|
meta.SetFolder(rootFolder) // Set the folder in the metadata to the root folder
|
|
} else {
|
|
var ok bool
|
|
fid, ok = r.folders.Tree().DirPath(folder, rootFolder)
|
|
if !ok {
|
|
// HACK: this is a hack to get the folder path without the root folder
|
|
// TODO: should we build the tree in a different way?
|
|
fid, ok = r.folders.Tree().DirPath(folder, "")
|
|
if !ok {
|
|
return "", fmt.Errorf("folder %s NOT found in tree", folder)
|
|
}
|
|
}
|
|
}
|
|
|
|
fileName := slugify.Slugify(title) + ".json"
|
|
if fid.Path != "" {
|
|
fileName = safepath.Join(fid.Path, fileName)
|
|
}
|
|
|
|
if options.Path != "" {
|
|
fileName = safepath.Join(options.Path, fileName)
|
|
}
|
|
|
|
parsed := ParsedResource{
|
|
Info: &repository.FileInfo{
|
|
Path: fileName,
|
|
Ref: options.Ref,
|
|
},
|
|
Obj: obj,
|
|
}
|
|
body, err := parsed.ToSaveBytes()
|
|
if err != nil {
|
|
return "", err
|
|
}
|
|
|
|
err = r.repo.Write(ctx, fileName, options.Ref, body, commitMessage)
|
|
if err != nil {
|
|
return "", fmt.Errorf("failed to write file: %s, %w", fileName, err)
|
|
}
|
|
|
|
return fileName, nil
|
|
}
|
|
|
|
func (r *ResourcesManager) WriteResourceFromFile(ctx context.Context, path string, ref string) (string, schema.GroupVersionKind, error) {
|
|
// Read the referenced file
|
|
readCtx, readSpan := tracing.Start(ctx, "provisioning.resources.write_resource_from_file.read_file")
|
|
fileInfo, err := r.repo.Read(readCtx, path, ref)
|
|
if err != nil {
|
|
readSpan.RecordError(err)
|
|
readSpan.End()
|
|
return "", schema.GroupVersionKind{}, fmt.Errorf("failed to read file: %w", err)
|
|
}
|
|
readSpan.End()
|
|
|
|
parseCtx, parseSpan := tracing.Start(ctx, "provisioning.resources.write_resource_from_file.parse_file")
|
|
parsed, err := r.parser.Parse(parseCtx, fileInfo)
|
|
if err != nil {
|
|
parseSpan.RecordError(err)
|
|
parseSpan.End()
|
|
return "", schema.GroupVersionKind{}, fmt.Errorf("failed to parse file: %w", err)
|
|
}
|
|
parseSpan.End()
|
|
|
|
if parsed.Obj.GetName() == "" {
|
|
return "", schema.GroupVersionKind{}, ErrMissingName
|
|
}
|
|
|
|
// Check if the resource already exists
|
|
id := resourceID{
|
|
Name: parsed.Obj.GetName(),
|
|
Resource: parsed.GVR.Resource,
|
|
Group: parsed.GVK.Group,
|
|
}
|
|
|
|
if existing, found := r.findResource(id); found {
|
|
return "", parsed.GVK, fmt.Errorf("duplicate resource name: %s, %s and %s: %w", parsed.Obj.GetName(), path, existing, ErrDuplicateName)
|
|
}
|
|
r.addResource(id, path)
|
|
|
|
// For resources that exist in folders, set the header annotation
|
|
if slices.Contains(SupportsFolderAnnotation, parsed.GVR.GroupResource()) {
|
|
// Make sure the parent folders exist
|
|
folderCtx, folderSpan := tracing.Start(ctx, "provisioning.resources.write_resource_from_file.ensure_folder")
|
|
folder, err := r.folders.EnsureFolderPathExist(folderCtx, path)
|
|
if err != nil {
|
|
folderSpan.RecordError(err)
|
|
folderSpan.End()
|
|
return "", parsed.GVK, fmt.Errorf("failed to ensure folder path exists: %w", err)
|
|
}
|
|
parsed.Meta.SetFolder(folder)
|
|
folderSpan.End()
|
|
}
|
|
|
|
// Clear any saved identifiers
|
|
parsed.Meta.SetUID("")
|
|
parsed.Meta.SetResourceVersion("")
|
|
|
|
runCtx, runSpan := tracing.Start(ctx, "provisioning.resources.write_resource_from_file.run_resource")
|
|
err = parsed.Run(runCtx)
|
|
if err != nil {
|
|
runSpan.RecordError(err)
|
|
}
|
|
runSpan.End()
|
|
|
|
return parsed.Obj.GetName(), parsed.GVK, err
|
|
}
|
|
|
|
func (r *ResourcesManager) RenameResourceFile(ctx context.Context, previousPath, previousRef, newPath, newRef string) (string, string, schema.GroupVersionKind, error) {
|
|
name, oldFolderName, gvk, err := r.RemoveResourceFromFile(ctx, previousPath, previousRef)
|
|
if err != nil {
|
|
return name, oldFolderName, gvk, fmt.Errorf("failed to remove resource: %w", err)
|
|
}
|
|
|
|
newName, gvk, err := r.WriteResourceFromFile(ctx, newPath, newRef)
|
|
if err != nil {
|
|
return name, oldFolderName, gvk, fmt.Errorf("failed to write resource: %w", err)
|
|
}
|
|
|
|
return newName, oldFolderName, gvk, nil
|
|
}
|
|
|
|
func (r *ResourcesManager) RemoveResourceFromFile(ctx context.Context, path string, ref string) (string, string, schema.GroupVersionKind, error) {
|
|
info, err := r.repo.Read(ctx, path, ref)
|
|
if err != nil {
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("failed to read file: %w", err)
|
|
}
|
|
|
|
obj, gvk, _ := DecodeYAMLObject(bytes.NewBuffer(info.Data))
|
|
if obj == nil {
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("no object found")
|
|
}
|
|
|
|
objName := obj.GetName()
|
|
if objName == "" {
|
|
return "", "", schema.GroupVersionKind{}, ErrMissingName
|
|
}
|
|
|
|
client, _, err := r.clients.ForKind(ctx, *gvk)
|
|
if err != nil {
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("unable to get client for deleted object: %w", err)
|
|
}
|
|
|
|
// the folder annotation is not stored in the git file, so we need to get it from grafana
|
|
grafanaObj, err := client.Get(ctx, objName, metav1.GetOptions{})
|
|
if err != nil {
|
|
if apierrors.IsNotFound(err) {
|
|
return objName, "", schema.GroupVersionKind{}, nil // Already deleted or simply non-existing, nothing to do
|
|
}
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("unable to get grafana object: %w", err)
|
|
}
|
|
meta, err := utils.MetaAccessor(grafanaObj)
|
|
if err != nil {
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("unable to get meta accessor: %w", err)
|
|
}
|
|
folderName := meta.GetFolder()
|
|
|
|
err = client.Delete(ctx, objName, metav1.DeleteOptions{})
|
|
if err != nil {
|
|
if apierrors.IsNotFound(err) {
|
|
return objName, folderName, schema.GroupVersionKind{}, nil // Already deleted or simply non-existing, nothing to do
|
|
}
|
|
|
|
return "", "", schema.GroupVersionKind{}, fmt.Errorf("failed to delete: %w", err)
|
|
}
|
|
|
|
return objName, folderName, schema.GroupVersionKind{}, nil
|
|
}
|