* Deprecate Legacy Storage Migration in Backend
* Change the messaging around legacy storage
* Disable cards to connect
* Commit import changes
* Block repository creation if resources are in legacy storage
* Update error message
* Prettify
* chore: uncomment unified migration
* chore: adapt and fix tests
* Remove legacy storage migration from frontend
* Refactor provisioning job options by removing legacy storage and history fields
- Removed the `History` field from `MigrateJobOptions` and related references in the codebase.
- Eliminated the `LegacyStorage` field from `RepositoryViewList` and its associated comments.
- Updated tests and generated OpenAPI schema to reflect these changes.
- Simplified the `MigrationWorker` by removing dependencies on legacy storage checks.
* Refactor OpenAPI schema and tests to remove deprecated fields
- Removed the `history` field from `MigrateJobOptions` and updated the OpenAPI schema accordingly.
- Eliminated the `legacyStorage` field from `RepositoryViewList` and its associated comments in the schema.
- Updated integration tests to reflect the removal of these fields.
* Fix typescript errors
* Refactor provisioning code to remove legacy storage dependencies
- Eliminated references to `dualwrite.Service` and related legacy storage checks across multiple files.
- Updated `APIBuilder`, `RepositoryController`, and `SyncWorker` to streamline resource handling without legacy storage considerations.
- Adjusted tests to reflect the removal of legacy storage mocks and dependencies, ensuring cleaner and more maintainable code.
* Fix unit tests
* Remove more references to legacy
* Enhance provisioning wizard with migration options
- Added a checkbox for migrating existing resources in the BootstrapStep component.
- Updated the form context to track the new migration option.
- Adjusted the SynchronizeStep and useCreateSyncJob hook to incorporate the migration logic.
- Enhanced localization with new descriptions and labels for migration features.
* Remove unused variable and dualwrite reference in provisioning code
- Eliminated an unused variable declaration in `provisioning_manifest.go`.
- Removed the `nil` reference for dualwrite in `repo_operator.go`, aligning with the standalone operator's assumption of unified storage.
* Update go.mod and go.sum to include new dependencies
- Added `github.com/grafana/grafana-app-sdk` version `0.48.5` and several indirect dependencies including `github.com/getkin/kin-openapi`, `github.com/hashicorp/errwrap`, and others.
- Updated `go.sum` to reflect the new dependencies and their respective versions.
* Refactor provisioning components for improved readability
- Simplified the import statement in HomePage.tsx by removing unnecessary line breaks.
- Consolidated props in the SynchronizeStep component for cleaner code.
- Enhanced the layout of the ProvisioningWizard component by streamlining the rendering of the SynchronizeStep.
* Deprecate MigrationWorker and clean up related comments
- Removed the deprecated MigrationWorker implementation and its associated comments from the provisioning code.
- This change reflects the ongoing effort to eliminate legacy components and improve code maintainability.
* Fix linting issues
* Add explicit comment
* Update useResourceStats hook in BootstrapStep component to accept selected target
- Modified the BootstrapStep component to pass the selected target to the useResourceStats hook.
- Updated related tests to reflect the change in expected arguments for the useResourceStats hook.
* fix(provisioning): Update migrate tests to match export-then-sync behavior for all repository types
Updates test expectations for folder-type repositories to match the
implementation changes where both folder and instance repository types
now run export followed by sync. Only the namespace cleaner is skipped
for folder-type repositories.
Changes:
- Update "should run export and sync for folder-type repositories" test to include export mocks
- Update "should fail when sync job fails for folder-type repositories" test to include export mocks
- Rename test to clarify that both export and sync run for folder types
- Add proper mock expectations for SetMessage, StrictMaxErrors, Process, and ResetResults
All migrate package tests now pass.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Update provisioning wizard text and improve resource counting display
- Enhanced descriptions for migrating existing resources to clarify that unmanaged resources will also be included.
- Refactored BootstrapStepResourceCounting component to simplify the rendering logic and ensure both external storage and unmanaged resources are displayed correctly.
- Updated alert messages in SynchronizeStep to reflect accurate information regarding resource management during migration.
- Adjusted localization strings for consistency with the new descriptions.
* Update provisioning wizard alert messages for clarity and accuracy
- Revised alert points to indicate that resources can still be modified during migration, with a note on potential export issues.
- Clarified that resources will be marked as managed post-provisioning and that dashboards remain accessible throughout the process.
* Fix issue with trigger wrong type of job
* Fix export failure when folder already exists in repository
When exporting resources to a repository, if a folder already exists,
the Read() method would fail with "path component is empty" error.
This occurred because:
1. Folders are identified by trailing slash (e.g., "Legacy Folder/")
2. The Read() method passes this path directly to GetTreeByPath()
3. GetTreeByPath() splits the path by "/" creating empty components
4. This causes the "path component is empty" error
The fix strips the trailing slash before calling GetTreeByPath() to
avoid empty path components, while still using the trailing slash
convention to identify directories.
The Create() method already handles this correctly by appending
".keep" to directory paths, which is why the first export succeeded
but subsequent exports failed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Fix folder tree not updated when folder already exists in repository
When exporting resources and a folder already exists in the repository,
the folder was not being added to the FolderManager's tree. This caused
subsequent dashboard exports to fail with "folder NOT found in tree".
The fix adds the folder to fm.tree even when it already exists in the
repository, ensuring all folders are available for resource lookups.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Revert "Merge remote-tracking branch 'origin/uncomment-unified-migration-code' into cleanup/deprecate-legacy-storage-migration-in-provisioning"
This reverts commit 6440fae342, reversing
changes made to ec39fb04f2.
* fix: handle empty folder titles in path construction
- Skip folders with empty titles in dirPath to avoid empty path components
- Skip folders with empty paths before checking if they exist in repository
- Fix unit tests to properly check useResourceStats hook calls with type annotations
* Update workspace
* Fix BootstrapStep tests after reverting unified migration merge
Updated test expectations to match the current component behavior where
resource counts are displayed for both instance and folder sync options.
- Changed 'Empty' count expectation from 3 to 4 (2 cards × 2 counts each)
- Changed '7 resources' test to use findAllByText instead of findByText
since the count appears in multiple cards
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
* Remove bubbletee deps
* Fix workspace
* provisioning: update error message to reference enableMigration config
Update the error message when provisioning cannot be used due to
incompatible data format to instruct users to enable data migration
for folders and dashboards using the enableMigration configuration
introduced in PR #114857.
Also update the test helper to include EnableMigration: true for both
dashboards and folders to match the new configuration pattern.
* provisioning: add comment explaining Mode5 and EnableMigration requirement
Add a comment in the integration test helper explaining that Provisioning
requires Mode5 (unified storage) and EnableMigration (data migration) as
it expects resources to be fully migrated to unified storage.
* Remove migrate resources checkbox from folder type provisioning wizard
- Remove checkbox UI for migrating existing resources in folder type
- Remove migrateExistingResources from migration logic
- Simplify migration to only use requiresMigration flag
- Remove unused translation keys
- Update i18n strings
* Fix linting
* Remove unnecessary React Fragment wrapper in BootstrapStep
* Address comments
---------
Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
* chore(deps): update nanogit to v0.3.0 in go.mod and go.sum files
* Add retry logic for nanogit client operations
- Configure retry logic in withGitContext to ensure all Git operations have retry support
- Use nanogit's ExponentialBackoffRetrier with 8 attempts (~10s retry window)
- Retry transient network errors and HTTP-specific server errors (5xx for GET/DELETE, 429 for all)
- Rename logger function to withGitContext to better reflect its responsibilities
* fix: resolve staticcheck S1008 linting issue in retry_client.go
Simplify return statement to use errors.As directly instead of if-return pattern
* Revert "fix: resolve staticcheck S1008 linting issue in retry_client.go"
This reverts commit bd367b5629.
* Provisioning: Improve logging and tracing in job processing
- Add comprehensive tracing with OpenTelemetry spans across all job operations
- Enhance logging with consistent style: lowercase, concise messages, appropriate log levels
- Use past tense for completed lifecycle events (e.g., 'stopped' vs 'stop')
- Add structured logging with contextual attributes for better searchability
- Handle graceful shutdowns without throwing errors on context cancellation
- Refactor Cleanup method into listExpiredJobs and cleanUpExpiredJob for better code quality
- Avoid double logging by only logging errors when handled locally
- Add tracing and logging to historyjob controller cleanup operations
Files modified:
- pkg/registry/apis/provisioning/jobs/driver.go: Add tracing spans and improve error handling for graceful shutdown
- pkg/registry/apis/provisioning/jobs/concurrent_driver.go: Add tracing and consistent logging
- pkg/registry/apis/provisioning/jobs/persistentstore.go: Add comprehensive tracing and logging to all public methods, refactor cleanup
- apps/provisioning/pkg/controller/historyjob.go: Add tracing and improve logging consistency
* Update pkg/registry/apis/provisioning/jobs/persistentstore.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Refactor logging in persistentstore.go
- Remove debug log statements at the start of job operations for cleaner output
- Maintain structured logging with contextual attributes for improved traceability
Files modified:
- pkg/registry/apis/provisioning/jobs/persistentstore.go: Clean up logging for job operations
* Enhance logging and tracing in provisioning job operations
- Introduce OpenTelemetry spans for better observability in job processing and webhook handling
- Improve structured logging with contextual attributes for key operations
- Remove unnecessary tracing spans in long-running functions to streamline performance
- Update error handling to record errors in spans for better traceability
Files modified:
- pkg/registry/apis/provisioning/controller/repository.go: Add tracing and structured logging to sync job operations
- pkg/registry/apis/provisioning/jobs/concurrent_driver.go: Remove tracing span from long-running function
- pkg/registry/apis/provisioning/jobs/driver.go: Enhance logging and tracing in job processing
- pkg/registry/apis/provisioning/webhooks/webhook.go: Implement tracing and structured logging for webhook connections
* Update pkg/registry/apis/provisioning/jobs/driver.go
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Improve error handling in ConcurrentJobDriver to differentiate between graceful shutdown and unexpected stops
* Remove unused import in driver.go to clean up code
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Validate Job Specs
* Add comprehensive unit test coverage for job validator
- Added 8 new test cases to improve coverage from 88.9% to ~100%
- Tests for migrate action without options
- Tests for delete/move actions with resources (missing kind)
- Tests for move action with valid resources
- Tests for move/delete with both paths and resources
- Tests for move action with invalid source paths
- Tests for push action with valid paths
Now covers all validation paths including resource validation and
edge cases for all job action types.
* Add integration tests for job validation
Added comprehensive integration tests that verify the job validator properly
rejects invalid job specifications via the API:
- Test job without action (required field)
- Test job with invalid action
- Test pull job without pull options
- Test push job without push options
- Test push job with invalid branch name (consecutive dots)
- Test push job with path traversal attempt
- Test delete job without paths or resources
- Test delete job with invalid path (path traversal)
- Test move job without target path
- Test move job without paths or resources
- Test move job with invalid target path (path traversal)
- Test migrate job without migrate options
- Test valid pull job to ensure validation doesn't block legitimate requests
These tests verify that the admission controller properly validates job specs
before they are persisted, ensuring security (path traversal prevention) and
data integrity (required fields/options).
* Remove valid job test case from integration tests
Removed the positive test case as it's not necessary for validation testing.
The integration tests now focus solely on verifying that invalid job specs
are properly rejected by the admission controller.
* Fix movejob_test to expect validation error at creation time
Updated the 'move without target path' test to expect the job creation
to fail with a validation error, rather than expecting the job to be
created and then fail during execution.
This aligns with the new job validation logic which rejects invalid
job specs at the API admission control level (422 Unprocessable Entity)
before they can be persisted.
This is better behavior as it prevents invalid jobs from being created
in the first place, rather than allowing them to be created and then
failing during execution.
* Simplify action validation using slices.Contains
Replaced manual loop with slices.Contains for cleaner, more idiomatic Go code.
This reduces code complexity while maintaining the same validation logic.
- Added import for 'slices' package
- Replaced 8-line loop with 1-line slices.Contains call
- All unit tests pass
* Refactor job action validation in ValidateJob function
Removed the hardcoded valid actions array and simplified the validation logic. The function now directly appends an error for invalid actions, improving code clarity and maintainability. This change aligns with the recent updates to job validation, ensuring that invalid job specifications are properly handled.
* adding some logs to better understand what might be happening
* only focus this PR on improve logging in finalizer handling
* debug log before calling finalizers
* working on finalizers
* removing last todos, adding unit tests
* better use SupportedFinalizers name
* addressing comments
* wip: fix tests and add delete error in status
* chore: codegen
* chore: codegen openapi
* Merge remote-tracking branch 'origin/main' into provisioning-finalisers-fix-2
* update frontend client
* fix: errors in testing
* fix: breaking test
---------
Co-authored-by: Daniele Ferru <daniele.ferru@grafana.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
* Configurable repository types in monolith and operator
* Default to Github in operators
* Regenerate wire
* Fix and implement unit tests
* Same types for enterprise tests
* Remove unnecessary conversion
* Remove the issue with import cycles
* Move repository package to apps
* Move operators to grafana/grafana
* Go mod tidy
* Own package by git sync team for now
* Merged
* Do not use settings in local extra
* Remove dependency on webhook extra
* Hack to work around issue with secure contracts
* Sync Go modules
* Revert "Move operators to grafana/grafana"
This reverts commit 9f19b30a2e.
* Add URLs to Job spec
* Rename them as RefURLs
* Implement RefURLs for Github
* Add Ref URLs to Jobs
* Worker Test
* Create the branch in the staged writer
* Regenerate Git mock
* Format code
* Consolidate ResourceURLs and RefURLs into one
* Fix broken tests
Roberto Jiménez Sánchez
Daniele Stefano Ferru
Gonzalo Trigueros Manzanas
Stephanie Hingtgen
Ryan McKinley
Costa Alexoglou
Charandas
Matheus Macabu