Commit Graph

36 Commits

Author SHA1 Message Date
Roberto Jiménez Sánchez 7e45a300b9 Provisioning: Remove migration from legacy storage (#112505)
* Deprecate Legacy Storage Migration in Backend

* Change the messaging around legacy storage

* Disable cards to connect

* Commit import changes

* Block repository creation if resources are in legacy storage

* Update error message

* Prettify

* chore: uncomment unified migration

* chore: adapt and fix tests

* Remove legacy storage migration from frontend

* Refactor provisioning job options by removing legacy storage and history fields

- Removed the `History` field from `MigrateJobOptions` and related references in the codebase.
- Eliminated the `LegacyStorage` field from `RepositoryViewList` and its associated comments.
- Updated tests and generated OpenAPI schema to reflect these changes.
- Simplified the `MigrationWorker` by removing dependencies on legacy storage checks.

* Refactor OpenAPI schema and tests to remove deprecated fields

- Removed the `history` field from `MigrateJobOptions` and updated the OpenAPI schema accordingly.
- Eliminated the `legacyStorage` field from `RepositoryViewList` and its associated comments in the schema.
- Updated integration tests to reflect the removal of these fields.

* Fix typescript errors

* Refactor provisioning code to remove legacy storage dependencies

- Eliminated references to `dualwrite.Service` and related legacy storage checks across multiple files.
- Updated `APIBuilder`, `RepositoryController`, and `SyncWorker` to streamline resource handling without legacy storage considerations.
- Adjusted tests to reflect the removal of legacy storage mocks and dependencies, ensuring cleaner and more maintainable code.

* Fix unit tests

* Remove more references to legacy

* Enhance provisioning wizard with migration options

- Added a checkbox for migrating existing resources in the BootstrapStep component.
- Updated the form context to track the new migration option.
- Adjusted the SynchronizeStep and useCreateSyncJob hook to incorporate the migration logic.
- Enhanced localization with new descriptions and labels for migration features.

* Remove unused variable and dualwrite reference in provisioning code

- Eliminated an unused variable declaration in `provisioning_manifest.go`.
- Removed the `nil` reference for dualwrite in `repo_operator.go`, aligning with the standalone operator's assumption of unified storage.

* Update go.mod and go.sum to include new dependencies

- Added `github.com/grafana/grafana-app-sdk` version `0.48.5` and several indirect dependencies including `github.com/getkin/kin-openapi`, `github.com/hashicorp/errwrap`, and others.
- Updated `go.sum` to reflect the new dependencies and their respective versions.

* Refactor provisioning components for improved readability

- Simplified the import statement in HomePage.tsx by removing unnecessary line breaks.
- Consolidated props in the SynchronizeStep component for cleaner code.
- Enhanced the layout of the ProvisioningWizard component by streamlining the rendering of the SynchronizeStep.

* Deprecate MigrationWorker and clean up related comments

- Removed the deprecated MigrationWorker implementation and its associated comments from the provisioning code.
- This change reflects the ongoing effort to eliminate legacy components and improve code maintainability.

* Fix linting issues

* Add explicit comment

* Update useResourceStats hook in BootstrapStep component to accept selected target

- Modified the BootstrapStep component to pass the selected target to the useResourceStats hook.
- Updated related tests to reflect the change in expected arguments for the useResourceStats hook.

* fix(provisioning): Update migrate tests to match export-then-sync behavior for all repository types

Updates test expectations for folder-type repositories to match the
implementation changes where both folder and instance repository types
now run export followed by sync. Only the namespace cleaner is skipped
for folder-type repositories.

Changes:
- Update "should run export and sync for folder-type repositories" test to include export mocks
- Update "should fail when sync job fails for folder-type repositories" test to include export mocks
- Rename test to clarify that both export and sync run for folder types
- Add proper mock expectations for SetMessage, StrictMaxErrors, Process, and ResetResults

All migrate package tests now pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Update provisioning wizard text and improve resource counting display

- Enhanced descriptions for migrating existing resources to clarify that unmanaged resources will also be included.
- Refactored BootstrapStepResourceCounting component to simplify the rendering logic and ensure both external storage and unmanaged resources are displayed correctly.
- Updated alert messages in SynchronizeStep to reflect accurate information regarding resource management during migration.
- Adjusted localization strings for consistency with the new descriptions.

* Update provisioning wizard alert messages for clarity and accuracy

- Revised alert points to indicate that resources can still be modified during migration, with a note on potential export issues.
- Clarified that resources will be marked as managed post-provisioning and that dashboards remain accessible throughout the process.

* Fix issue with trigger wrong type of job

* Fix export failure when folder already exists in repository

When exporting resources to a repository, if a folder already exists,
the Read() method would fail with "path component is empty" error.

This occurred because:
1. Folders are identified by trailing slash (e.g., "Legacy Folder/")
2. The Read() method passes this path directly to GetTreeByPath()
3. GetTreeByPath() splits the path by "/" creating empty components
4. This causes the "path component is empty" error

The fix strips the trailing slash before calling GetTreeByPath() to
avoid empty path components, while still using the trailing slash
convention to identify directories.

The Create() method already handles this correctly by appending
".keep" to directory paths, which is why the first export succeeded
but subsequent exports failed.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Fix folder tree not updated when folder already exists in repository

When exporting resources and a folder already exists in the repository,
the folder was not being added to the FolderManager's tree. This caused
subsequent dashboard exports to fail with "folder NOT found in tree".

The fix adds the folder to fm.tree even when it already exists in the
repository, ensuring all folders are available for resource lookups.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Revert "Merge remote-tracking branch 'origin/uncomment-unified-migration-code' into cleanup/deprecate-legacy-storage-migration-in-provisioning"

This reverts commit 6440fae342, reversing
changes made to ec39fb04f2.

* fix: handle empty folder titles in path construction

- Skip folders with empty titles in dirPath to avoid empty path components
- Skip folders with empty paths before checking if they exist in repository
- Fix unit tests to properly check useResourceStats hook calls with type annotations

* Update workspace

* Fix BootstrapStep tests after reverting unified migration merge

Updated test expectations to match the current component behavior where
resource counts are displayed for both instance and folder sync options.

- Changed 'Empty' count expectation from 3 to 4 (2 cards × 2 counts each)
- Changed '7 resources' test to use findAllByText instead of findByText
  since the count appears in multiple cards

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* Remove bubbletee deps

* Fix workspace

* provisioning: update error message to reference enableMigration config

Update the error message when provisioning cannot be used due to
incompatible data format to instruct users to enable data migration
for folders and dashboards using the enableMigration configuration
introduced in PR #114857.

Also update the test helper to include EnableMigration: true for both
dashboards and folders to match the new configuration pattern.

* provisioning: add comment explaining Mode5 and EnableMigration requirement

Add a comment in the integration test helper explaining that Provisioning
requires Mode5 (unified storage) and EnableMigration (data migration) as
it expects resources to be fully migrated to unified storage.

* Remove migrate resources checkbox from folder type provisioning wizard

- Remove checkbox UI for migrating existing resources in folder type
- Remove migrateExistingResources from migration logic
- Simplify migration to only use requiresMigration flag
- Remove unused translation keys
- Update i18n strings

* Fix linting

* Remove unnecessary React Fragment wrapper in BootstrapStep

* Address comments

---------

Co-authored-by: Rafael Paulovic <rafael.paulovic@grafana.com>
Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-17 17:22:17 +01:00
Alexander Zobnin e2d12f4cce Zanzana: Refactor remote client initialization (#114142)
* Zanzana: Refactor remote client

* rename config field URL to Addr

* Instrument grpc queries

* fix duplicated field
2025-12-11 10:55:12 +01:00
Costa Alexoglou bbfb8268d1 Provisioning: concurrent deletes in finalizers and 404 handling (#113155)
* fix: concurrent deletes in finalizers and 404 handling

* chore: feedback review

* fix: broken tests
2025-10-30 11:55:36 +01:00
Costa Alexoglou 4959197e37 fix: speedup client (#112640) 2025-10-21 10:48:43 +02:00
Matheus Macabu 93af167f9c Secrets: Add config for enabling gRPC client-side load balancing (#111989) 2025-10-20 14:04:59 +02:00
Stephanie Hingtgen 3ce9137c19 Provisioning: Refactor to combine validation and test endpoint logic (#111965)
Provisioning: Refactor test endpoint
2025-10-03 01:16:37 -06:00
Daniele Stefano Ferru e34475041a Provisioning: move job operator to enterprise and enable PR worker (#111663) 2025-09-27 20:14:08 -05:00
Stephanie Hingtgen e53b5902fa Provisioning: Wire up tracing and add trace for job sync (#111455) 2025-09-22 22:23:29 -06:00
Stephanie Hingtgen 8b1caccc72 Provisioning: Add metrics for repo controller (#111450) 2025-09-22 20:14:03 +00:00
Stephanie Hingtgen a2d09490ae Provisioning: Allow setting concurrent jobs settings and add metric (#111451) 2025-09-22 20:07:22 +00:00
Stephanie Hingtgen b02d2762f5 Provisioning: Add metrics to jobs (#111447) 2025-09-22 19:05:34 +00:00
Stephanie Hingtgen bd550d2f06 Provisioning: Wire up prometheus (#111444) 2025-09-22 09:54:50 -05:00
Mihai Turdean aaa8094a53 Add more metrics to the IAM Folder Reconciler (#111275)
* Add more metrics to the operator

* Add namespace when logging metrics

* Skip flaky test
2025-09-18 09:03:32 -06:00
Mihai Turdean ac13da2d1d Use concurrent informer for iam-folder-reconciler (#110987) 2025-09-11 15:57:07 -05:00
Mihai Turdean 03abe18bb2 Instrument IAM Folder Operator with Tracing (#110923) 2025-09-11 15:21:44 +00:00
Mihai Turdean 338fc84479 Enable metrics by default for the folder operator (#110852) 2025-09-10 08:55:28 -06:00
Roberto Jiménez Sánchez acbc2cf01a Provisioning: Configurable Repository Types in monolith and operators (#110822)
* Configurable repository types in monolith and operator

* Default to Github in operators

* Regenerate wire

* Fix and implement unit tests

* Same types for enterprise tests

* Remove unnecessary conversion

* Remove the issue with import cycles
2025-09-09 19:13:22 +02:00
Daniele Stefano Ferru 76976ef648 Provisioning: add webhook support in API Server and Operator (#110673)
* Provisioning: add webhook support in API Server

* updating Extra interface

* adding extra with workers interface

* reverting extraWithWorkers in RegisterAPIService

* adding extra job worker provider

* adding new extra job provider

* Wire things differently

* Remove unused GetJobs

* Pass url variable as string

* Support webhooks in controller

* Fix condition

* Change the naming

---------

Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>
2025-09-08 19:39:05 +02:00
Roberto Jiménez Sánchez ace05e999d Provisioning: Remove temporary logic to test clients in jobs operator (#110758)
Remove temporary logic to test clients in jobs operator
2025-09-08 12:15:13 +00:00
Daniele Stefano Ferru 76f7836419 Provisioning: correctly use resource clients in controllers (#110737)
* Provisioning: correctly use resource clients in controllers

* better names on fields

* fix struct initialisation

* updating roundtripper tests
2025-09-06 18:13:39 -06:00
Daniele Stefano Ferru 04c3d9bff1 Provisioning: fixing some checks in config (#110729) 2025-09-05 23:27:35 +00:00
Mihai Turdean 62cc0f9c0e Udate IAM Folder Reconciler Operator config (#110728) 2025-09-05 22:56:23 +00:00
Stephanie Hingtgen 2750a3516a Provisioning: Add provisioning api server to clients (#110721) 2025-09-05 20:36:52 +00:00
Roberto Jiménez Sánchez ed2273b2d2 Provisioning: processing of jobs in job controller (#110223)
* WIP: Controller

* WIP: more changes

* Use patcher from new location

* Separate import

* Move operators to grafana/grafana

* Tidy go mod

* Remove duplicate TODO

* Wrapper for unified storage

* WIP: build unified storage client

* More attempts

* Revert update workspace

* Improve comment

* Fix linting

* Change signature of repository getter

* Add ticket numbers

* Remove question

* Read config from file for decrypt service

* Config struct for unified torage

* Add local config

* Fix compilation

* Try to configure it

* Fix linting

* Add FIXME comment

* Move reusable logic into controller config

* Remove unused

* More logic to be reused

* Extract workers into separate function

* Clean up unified storage client

* Revert a couple of files

* Remove secrets decrypter from this PR

* Revert enterprise imports

* Clean up unified storage setup logic

* Add TODO

* Revert some changes

* Remove file

* Use the expected clients

---------

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-05 18:28:31 +00:00
Roberto Jiménez Sánchez e2913815d3 Provisioning: Build resource clients for operators (#110699)
---------

Co-authored-by: Stephanie Hingtgen <stephanie.hingtgen@grafana.com>
2025-09-05 17:52:57 +00:00
Stephanie Hingtgen c4bc83019d Provisioning: Fix bug in config reading (#110703) 2025-09-05 14:30:43 +00:00
Roberto Jiménez Sánchez feb4368de5 Provisioning: Build unified storage client for operators (#110671)
* Build unified storage client in config
* Testing ticker to count managed objects
2025-09-05 10:51:51 +00:00
Stephanie Hingtgen 9ddc70423b Provisioning: Cleanup tester interface (#110640)
* Provisioning: Cleanup tester interface

* undo accidental change

* cleanup

* cleanup test
2025-09-05 07:47:27 +02:00
Stephanie Hingtgen b567cde3d3 Provisioning: Reuse controller from registry (#110639) 2025-09-05 01:13:55 +00:00
Mihai Turdean f1dffca140 IAM Folder Reconciler: Make operator available as a target when running the grafana binary. (#110567) 2025-09-04 08:22:17 -06:00
Roberto Jiménez Sánchez 57ea65dc42 Provisioning: Decrypt using Secrets Service in Operators (#110589)
* Use real secrets servicer decrypter

* Add also server name just in case
2025-09-04 16:18:14 +02:00
Roberto Jiménez Sánchez 3d009ff7ed Provisioning: Build and use repository factory in repository controller (#110585)
Build and use repository factory
2025-09-04 13:12:56 +02:00
Stephanie Hingtgen 4527530cd2 Provisioning: update readme to skip migrations (#110561) 2025-09-03 21:09:44 +00:00
Stephanie Hingtgen 84ae9ea71b Provisioning: Add scaffolding for repo controller (#110543) 2025-09-03 17:30:41 +00:00
Stephanie Hingtgen 8e41d5929a Operators: Cleanup registration process (#110539) 2025-09-03 16:36:42 +00:00
Roberto Jiménez Sánchez 4de9ec7310 Provisioning: Fix import cycle between grafana and provisioning app (#110406)
* Move operators to grafana/grafana

* Go mod tidy
2025-09-01 13:29:34 +00:00