public/grafana
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

AppPlatform: Introduce experimental Github integration for dashboard configuration management (#96329)

Browse Source 
* [Provisioning] Pay back some technical debt (#100720)

* Handle pagination in github client

* Add some unit test coverage

* Remove unknown repository

* Remove unknown leftover

* Revert "Add some unit test coverage"

This reverts commit 420c9674d2.

* Revert "Revert "Add some unit test coverage""

This reverts commit f7eca41957.

* Revert unit tests in github package

* Remove S3 case as it's now deprecated

* [Provisioning] Consolidate job status report in JobProgressRecorder (#100718)

* Log also successful operation

* Consolidate stop logic under TooManyErrors

* Use error for TooManyErrors

* Pass the progress recorder

* Define JobProgressRecorder interface

* Do not expect workers to return status

* Remove scenarios due to pointers

* Use recorder to manage the entire state

* Provisioning: Support rotating secrets (#100705)

* Provisioning: Refactor webhook to another interface (#100733)

* POC/Provisioning: Remove S3 references (#100734)

* Remove unused script

* Remove s3 references

* Provisioning: Keep the existing k8s name if it is specified in metadata (#100672)

* keep name

* keep name

* Revert "keep name"

This reverts commit 29f87bcaeb.

* Commit stale go.mod

* Keep name also for sync deletions

---------

Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>

* Update preview banner copy

* Query Library: Move backend to enterprise (#100371)

* Fix wire

* Fix jobs table re-renders

* Provisioning: Refactor history to its own interface (#100735)

* Provisioning: Refactor history to its own interface

* refactor: use VersionedRepository

* Update API

* Provisioning: Test the GitHub client directly (#100808)

* Provisioning: Test the GitHub client directly

Instead of mocking the abstracted client, test it as well by mocking the underlying GitHub client. This also lets us
remove the mock for the abstracted client.

* refactor: move out helpers

* chore: set dependency owner

* Provisioning: Better clone/push error support (#100854)

* Provisioning: Replace searcher with one that knows about modes (#100857)

* Provisioning: Start in "mode5" when nothing exists in legacy (#100862)

* [Provisioning] Fix duplicate sync jobs triggered in controller (#100870)

* Improve logging on reasons why the controller triggered

* Fix messaging for sync job

* fix lint

* Provisioning: Move legacy export/import into a single migrate job (#100865)

* [Provisioning] Miscellanenous bug fixes and improvements (#100976)

* Error if found duplicate ID

* Fix issue with manual test button

* Fix issue with health errors not going away

* Display status in sync overview

* Use patch operations instead

* Trigger sync job after status update

* Convert Export Tab into modal

* Remove unused FieldSet import

* Only last 8 jobs

* Remove Links card

* Use button for Github Source Code

* Add actions to resources page

* Add resource column to Repository Resources

* Display Job Spec in RecentJobs

* Display dates in history page

* Display Avatar if available

* Improve styling of the avatar

* Update betterer

* Remove duplicate history header in history

* Commit betterer

* Address code styling issues

* update flags

* github v69

* v69

* POC/Provisioning: Add wizard (#100596)

* Chore: make update-workspace

* Chore: Fix lints (#101039)

* Provisioning: Workflows as write access (#101031)

* workflow as write access

* workflow as write access

* workflow as write access

* Update pkg/registry/apis/provisioning/repository/test.go

Co-authored-by: Mariell Hoversholm <mariell.hoversholm@grafana.com>

* POC/Provisioning: Add wizard (#100596)

* update refs

* update refs

* lint fix

* lint fix

* lint fix

* default everythign to read only

* reuse form components

* remove main

---------

Co-authored-by: Mariell Hoversholm <mariell.hoversholm@grafana.com>
Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* merge main

* Fix workflow types

* Betterer

* [Provisioning] Fix webhook and finalizer issues (#101052)

Fix webhook and finalizer issue

The maximum number of webhooks per repository is 20

* [Provisioning] Fix issue with last ref (#101056)

* Fix issue with last ref

* Update frontend code

* Fix the local tmp test

* Use lastRef

* POC/Provisioning: Simplify connect step (#101064)

* Fix sending workflows

* Use write for local

* Move connect action to the next step

* Remove wizard props

* Typo

* Redirect to wizard

* Show repo link after successful export

* Provisioning: Avoid starting sync jobs when using legacy storage (#101114)

* avoid starting sync jobs on legacy

* newlines

* Provisioning: Onboarding landing page (#101112)

* add landing page before wizard

* Update onboarding page

* Update URL

* Remove unused

* Add deleteAll button

* Improved text

* betterer

---------

Co-authored-by: Clarity-89 <homes89@ukr.net>

* Provisioning: use the sync job to finish the migrate job (#101107)

* Provisioning: Show progress more often (#101128)

* show progress bar earlier

* show progress bar earlier

* update wording to be less specific

* POC/Provisioning: Enable sync (#101131)

* update preview banner

* actualy remove and don't crash without provisioning flag

* Update db banner

* Provisioning: Export oldest items first (#101189)

* Provisioning: better branch handling (#101188)

* add missing file

* Provisioning: Fix tests (#101197)

* Provisioning: Refactor tests to be multiple functions

* Provisioning: Fix tests

* fix: make github-example sync

* fix misspell

* Provisioning: avoid migration wizard if things are already in unified storage (#101204)

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* more lint

* POC/Provisioning: Handle connect step errors (#101192)

* Provisioning: Unify status handling in repository and migrate steps

* Refactor: Move WorkflowsField outside RepositoryStep and use proper type

* Refactor: Improve repository verification error handling and UI

* Refactor: Simplify repository verification error handling

* Refactor: Simplify RepositoryStep component structure

* Refactor: Improve error handling in RepositoryStep

* Refactor: Remove redundant repository creation logic from ProvisioningWizard

* Refactor: Simplify RequestErrorAlert component

* show github error

* now will verify

* test .git

* recover from bad config

* Update error handling

* Remove unused prop

* merge upstream

* Show migration summary

* Update text

* Improve text

* Betterer

* [Provisioning] Review controller changes (#101216)

* Review health check conditions

* Move down the logic to set up the sync status

* Skip if it's only a health check rerun

* Fix health check conditions

* Preserve last ref

* Format code

* Rename to shouldSkipSync

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Roberto Jiménez Sánchez <roberto.jimenez@grafana.com>

* fix promotion step

* In the promotion pipeline, publish should depend on build

* fix promotion pipeline

* [Provisioning] Use smaller methods to process repository events (#101240)

* update codegen

* merge main

* Provisioning: Avoid localhost error loop (#101253)

* Provisioning: Update the recent jobs formatting (#101250)

format history

* [Provisioning] Refactor Pull Request & Lint worker (#101273)

* Refactor the code

* Refactor into separate files

* Consolidate linter flag in one spot

* Use global feature flags

* Commit betterer

* Remove from JSON the intermidiate flag

* Use again spec

* Clean up

* Revert changes in test

* POC/Provisioning: Remove sync confirm modal (#101281)

* [Provisioning] Remove linting from MVP (#101286)

* Remove Linting backend

* Re-generate client

* POC/Provisioning: Unify tags (#101218)

* Unify tags

* add both tags

* add tag types

* Check for the redirect only once

* Add fetch settings with delay hook

* Refetch settings

* Split hooks into separate files

* Cleanup

* Prettier

* Prettier

* Remove lint code

* Betterer

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Provisioning: Update token instructions (#101280)

* Provisioning: Use blob storage rather than local file system to save images (#101298)

* [Provisioning] Add more explicit setup warnings if webhook integration and image rendering are disabled (#101304)

* Setup warnings if renders or webhooks are not possible

* Improve display

* Use a single Alert

* Make design more compact

* Only display local config is missing critical feature toggles

* Improve styling of required flags

* Add file name to the custom ini itself

* Add copy button

* Add FeatureSection Component

* Commit betterer

* Use an interactive table

* Use a modal for instructions

* Use the same modal for example config

* Improve setup steps

* Improve stepper

* Copy code ctrl + c

* Make it more compact

* Select feature to enable

* Improve the height of alerts

* Separate components

* Better warnings

* Improve the page

* Improve the cards

* Improve cards even more

* Improve cards

* Improve cards

* Optional copy in code block

* Add side bar with steps

* Improve styling

* Style modal

* Clean up Code

* Remove index file

* Simplify Step Component

* Commit betterer

* Simplify components

* Use CodeEditor and Clipboard components

* Do not show scrollback on minimum size

* Fix positioning of footer

* Separate Component for Feature

* Use different styling

* Commit betterer

* Use more Grafana components in the FeatureCard

* Separate sidebar into own components

* Simplify sidebar code

* Commit betterer

* Remove connector

* Simplify styling further

* Use cards

* Improve code

* Use more grafana component in InstructionsModal

* Further simplify

* Simplify the code

* Simplify style

* Clean up

* Simplify the Wizard

* Use little icons

* Improve feature cards

* Improve cards

* Commit betterer

* Add description to feature setup

* Improve instructions for snapshot preview

* Move all files into Setup folder

* Commit betterer

* Clean up the warnings code

* Improve coding

* Move sidebar item to separate fiel

* Rename components

* Fix issues

* Use stack instead

* Improve style

* Don't show setup button if configured already

* Simplify again CSS

* Use secondary actions

* Style a bit more

* Improve wording

* Update warning

* Refer to docs in Image Renderer

* More clean up

* Revert changes in generated client

* Fix typos and imports

* Fix lint errors

* Provisioning: better error support (#101490)

* update openapi snapshot

* fix build

* Provisioning: Only show setup page when feature toggles are missing (#101502)

* form fixing

* form fixing

* always send UID

* Same onboarding page regarless of migration (#101557)

* backend building... frontend still broken

* rename sync with main

* Provisioning: Update dashboard badge (#101599)

* Rename to push / pull everything user-facing (#101577)

* Rename to push / pull all everything user-facing

* Use automatic pulling wording

* Provisioning: Migrate when using unified storage (#101572)

* migrate when not unified

* Update pkg/registry/apis/provisioning/register.go

Co-authored-by: Roberto Jiménez Sánchez <roberto.jimenez@grafana.com>

* variables

* merge main

---------

Co-authored-by: Roberto Jiménez Sánchez <roberto.jimenez@grafana.com>

* Add tabs and features tab to listing page (#101570)

* List features in onboarding page (#101558)

* merge main

* POC/Provisioning: Check if the instance is provisioned (#101601)

* Check if the instance is provisioned

* Fix lints

* Fix getting config for new dashboard

* Fixes after merge

* More fixes

* Show success message

* Fix default value

* Add test

* Fix lints

* Provisioning: Include URLs in ResourceWrapper response (#101511)

* Convert Migrate wizard into a Connection Wizard (#101575)

* Convert Migrate wizard into a Connection Wizard

* Remove duplicate empty state

* Allow users to select target in the first step

* Remove file created by merge

* Select target based on existing connections

* Default option for targets and explainatory alert

* Do not display connect button if single connection

* Display target as tag in repository card

* Add Pull Step

* Fix linting

* User decides if migrate or connect

* Improve style based based on review

* Provisioning: Return upsert resource when writing (#101574)

* [Provisioning] Getting Started Page and Tab (#101701)

* merge main

* fix go.mod

* Provisioning: Redirect to the new URL after save (#101757)

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* POC/Provisioning: create folder (#101619)

* Add NewProvisionedFolderForm

* Cleanup

* Add folder api

* Register API

* Do not show provisioned badge for instances

* Enable saving

* Show saved alert

* Fixes

* Fix deps

* Cleanup

* Add test

* Add test

* Updates

* Fix test

* Fix import

* [Provisioning] Display instance connection directly in home (#101720)

* Show tabs directly in home page for instance sync

* Display actions also in the home page

* Add delete button to actions

* Fix issue with files tab

* Display tabs also in instant sync

* Generate connection name for instance sync

* Fix issues when no repositories

* Set active tab

* Remove leftover

* Rename component for listing folder repository

* Fix linting issues

* Fix merge error

* Rename to HomePage

* Split folder list into separate component

* Create own component for repository card

* Improve RepositoryCard readability

* Improve RepositoryCard styling

* Make view primary button

* Fix syntax

* Fix generateName

* merge main

* Provisioning: Move folder management into its own helper (#101864)

* folder

* update folder links

* Fix test

* update

* cleanup

---------

Co-authored-by: Clarity-89 <homes89@ukr.net>

* fixed folder issue

* POC/Provisioning: Create folder from root (#101921)

* Enable creating folder at the root

* Fix test

* Add interceptor

* Provisioning: Expose stats (#101927)

* [Provisioning] Unified onboarding wizard (#101952)

* Spike the solution

* More work

* Add more situations

* Attempt to display count of dashboards and folders

* Attempt with file count

* Do not display options if not possible

* Improve styling resources

* Use another API

* Fix issue with selection

* Style a bit

* Fix more issues

* Make the sync step work

* Improve links

* Use LinkButton

* Start pull automatically

* Start migration automatically

* Fix issue with options

* Fix issues

* Fix loading error

* Improve more things

* Improve styling

* Improve messaging

* Set the autofocus

* Fix some issues

* Fix issue with disabled options

* Only resources

* Finish settings depending on configuration

* Move title to wizard

* Fix title

* Improve styling

* Badge

* Explain on hover

* Improve styling

* Disabled at the bottom

* History & identifiers

* Improve wording

* Add padding left and right disable options

* Delete repository

* Improve buttons

* Give index time to catch up

* Improve buttons

* Handle steps with only forms

* Fix issue with initial migrate or pull

* Commit betterer

* Error messages

* Use memo

* Revampt that a bit

* Attempt to simplify the state and components

* Improve the component for Migrate

* Commit betterer

* Fix issue in next button

* Clean up more

* Start for boostrap step

* Fix issue with running status

* Fix issue with loading bootstrapping

* Improve loading

* Improve more the loading

* Fix issue with loading

* Empty tree

* Handle error

* Fix issue with looping

* Remove commented out lines

* Add comment

* Remove accidental file

* Fix imports

* Improve MigrateStep and PullStep

* Use hook for step status

* JobStep component

* Refactor data fetching

* Validate with Github

* Fix issue with failed error

* Fix next on success

* Address small comments

* Separate file for WizardContent

* Fix linting

* Use step approach also for bootstrap

* Make the logic for moving between steps clearer

* Fix navigation issue

* Clean up some logic

* Use useAsync for JobStep steps

* Revert "Use useAsync for JobStep steps"

This reverts commit 242a275cc9.

* Provisioning: use service to get counts (#101972)

counts

* must migrate when using legacy storage

* Revert "Revert "Use useAsync for JobStep steps""

This reverts commit a420d0ac36.

* Fix async conditions

* Organize imports

* Separate component for BootstrapOptionCard

* BootstrapOptionsList

* Remove duplicate definitions

---------

Co-authored-by: Clarity-89 <homes89@ukr.net>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Merge

* Fix utils

* POC/Provisioning: Fix folder path for dashboard (#101997)

* Provisioning: Fix folder path for dashboards

* Fix isNew

* Update test

* Fix any error

* Betterer

* [Provisioning] Improve progress recording and updates (#102035)

* do not validate on delete

* Provisioning: Implement authorizer for remaining resources (#101945)

* feat: implement authorizer for remaining resources

* fix: don't allow viewers to write files

* security: harden blob id fetching

* add integration test for admin vs viewer

* feat: only Get is a valid verb for reads in our subresources

Co-Authored-By: Ryan McKinley <ryantxu@gmail.com>

* feat: allow render for all requests

* refactor: use guards

Not changing code that goes `if a { } else if b { } else { }` as the semantic meaning of the different branches is
easier to parse.

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* POC/Provisioning: Improve wizard setup (#102066)

* wizard actions

* workign better

* remove more memo

* show polling interval

* cleanup

* finalizers

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/WizardContent.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/WizardContent.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* Update public/app/features/provisioning/Wizard/BootstrapStep.tsx

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>

* updates from alex

* updates from alex

* Simplify actions

* Extract props

* history supported form legacy only (for now)

* More refactor

* change order

* Fix cleanup finalizer

* show kinds

* fix lint

---------

Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>
Co-authored-by: Clarity-89 <homes89@ukr.net>
Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>

* Fix resource condition (#102086)

* Fix job summary stats issue (#102084)

* merge main

* Repository link should point to configured branch (#102092)

* Add getRepoHref

* Fix random string generation

* Redirect to Home on repository deletion (#102096)

* Fix extra commas in pull request comment (#102108)

* Fix image rendering endpoint (#102107)

* POC/Provisioning: Support migrate... when starting with unified storage (#102097)

* use same clone

* now using upsert

* Fix lint

---------

Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>

* Clean up unprovisioned resources after unified storage migration (#102126)

* Clean up unprovisioned resources after migrate

* Clean up unprovisioned resources after migrate

* Update pkg/registry/apis/provisioning/jobs/migrate/resources.go

* Reset summary between export and pull (#102101)

* Reset summary between export and pull

* Add reset results to unified storage migration

* Provisioning: always dirty (#102151)

* fix test version

* log the watch line

* POC/Provisioning: Disable repository list watch (#102169)

* Disable watch for repo list endpoint

* Add comment

* Remove another watch

* Provisioning: Avoid calling test on every update (#102161)

test less often

* Provisioning: Support prefixes in GitHub repositories (#101969)

* feat: add a Prefix property to GitHub repo spec

* feat: make nested folders work properly

* feat: use subdir for go-git export

* fix: placeholder for prefix should be grafana/

* feat: rename prefix to path

* fix: json name should be path, too

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* chore: regen apis

* fix: copy 'path'

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Merge

* Fix duplicate type

* Provisioning: Test export functionality (#101336)

* Provisioning: Test export functionality

* fix: use context.Background

* test: add more cases for local repo path resolving

* test: rework test inputs

* test: try to make github test work

* fix: clear global state

* Update api client imports

* Fix import

* Fix test

* Update codegen

* Provisioning: Make it green (#102271)

* chore: remove unused functions

* chore: update betterer results

* chore: update openapi spec

* chore: yarn generate-apis

* fix: specify default false if undefined

* Use AnnoKeyManagerIdentity

* Add manager kind

* POC/Provisioning: Update component structure (#102297)

* Update project structure

* Update imports

* Remove unused components

* Copy fixes

* Typo

* More copy fixes

* Betterer

* Update test

* merge main

* Provisioning: Replace hardcoded clients with discovery client (#101918)

* disco client

* discovery client

* merge main

* merge main

* keep factory

* keep factory

* find preffered version for delete factory

* use same folders request

* merge main

* with integration test

* POC/Provisioning: Compare spec in test rather than raw JSON (#102352)

* compare spec not json

* compare spec not json

* [Provisioning] Add in-code TODOs in API Server area (#102360)

* Add TODOs for files endpoint

* Add TODO history endpoint

* Add TODO to move files logic to resource package

* Add TODO to not use private fields directly

* Remove unnecessary checks in list connector

* Add pagination TODO in lister

* Add TODO to rename resources

* Add todo about cloning too early

* Add TODO to propose to merge sync and migrate endpoints

* Add TODOs in register

* Add more TODOs in connectors & routes

* Add TODOs about prefix

* Change it to remove

* Update pkg/registry/apis/provisioning/test.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* POC/Provisioning: Switch folders to use managedBy (#102362)

* Folders: Switch to managedBy

* Fix create folder

* Fix tests

* Do not allow changing folder from dashboard settings

* Update imports

* Update provisioned meta

* Do not show provisioned badge for child folders

* Fix folder title

* Update folder actions

* Update new provisioned folder form

* Remove unused code

* Fix condition

* Reset default values on change

* Remove duplicate nav item

* Add managedBy to DashboardQueryResult

* Provisioning: support watch over live (in feature branch) (#102408)

* Provisioning: watch cleanup (#102424)

* fix lint

* Provisioning: Add basic usage stats (#102405)

* [Provisioning] Add limitations to Github Repository (#102451)

* Put limits to Github

* File is too large

* Move constants

* Embed ListOptions again

* Remove TODO

* Provisioning: Pick a better default title (#102516)

better title

* Provisioning: sanitize pull request urls (#102517)

* [Provisioning] Clean up clone after export and migrate (#102467)

* Remove clone directory on clone failure

* Defer remove clones

* Log error if removal fails

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* [Provisioning] Limit path length and depth in APIs (#102472)

* Limit filepath length in files API calls

* Add common utility to deal with paths

* Use the existing function

* Fix import

* Update pkg/registry/apis/provisioning/safepath/limit.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Fix issue after website commit

* Fix linting issue in test

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Add timeout, JSON check and max body size to endpoints (#102443)

* Add timeout, JSON check and max body size to endpoints

* Use http.MaxBytesReader instead

* Use MaxBytesReader also for reading the entire body

* Add empty line

* Add unit tests

* Fix integration tests

* Update pkg/registry/apis/provisioning/render.go

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* Do not use utils :)

* Fix comment on unmarshalJSON

* 25MB for webhook events

* Remove content type check for files write

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* fix imports

* Provisioning: Remove export option from the UI (#102511)

* [Provisioning] Remove unused checkout method in go-git (#102460)

* [Provisioning] Limit max number of repositories to 10 (#102542)

* Limit to maximum 10 repositories in backend

* Change messaging

* Do not display connect button if more than 10

* Only fetch settings once

* watch repos

---------

Co-authored-by: Ryan McKinley <ryantxu@gmail.com>

* [Provisioning] Limit size and time to git clone and push in go-git (#102458)

* Limit git clone and push time and size

* Fix linting

* Use transport instead to limit

* Remove not supported

* Add TODO to make timeout configurable

* chore: make update-workspace

* Provisioning: Implement a new job queue (#102446)

* feat: implement a new job queue

Outstanding problems:

  * Status isn't saved.
  * Progress updates don't work (due to status not being saved probably?).

* feat: properly save status

* chore: document label

* chore: assumptions do hold

* fix: support multi-tenant job drivers

* fix: use namespace=*

* fix: set resource back to pointer when updating job progress

If we don't do this, we start rejecting job progress updates as the version falls out of sync.

* feat: make job APIs read-only

* fix: complete job when worker returns

* fix: set namespace on requests from controller

* test: check historic jobs

* chore: regen apis

* feat: start augmenting frontend

* feat: add jobs to authorizer

* feat: use watch from input

* fix: make frontend subscribe to historic jobs

* fix: lint

* chore: yarn prettier:write

* fix: frontend lints

* test: allow for empty state in historicjobs

* test: set content type for export request

* fix: always set job name on insert

* fix: import

* fix: use dashes not colons

* fix: job status should expect a historic job transition

* fix: allow PR jobs from multiple PRs

* feat: same name for sync and migrate jobs

* feat: generate a job name in the store

* refactor: rename to persistentStore

* feat: remove status subresources on jobs

* feat: join jobs into one card

* chore: regen openapi snapshot

---------

Co-authored-by: Roberto Jimenez Sanchez <roberto.jimenez@grafana.com>

* Provisioning: Use a complete storage for jobs (#102605)

* feat: add a complete strategy to apiserver

* feat: use the complete storage strategy for jobs

* test: behaviour changed in main

* [Provisioning] Consolidate file path handling (#102617)

* Add more cases for validation

* Call the method dir

* Clean files endpoint

* Simplify further

* Fix issues with folder

* Add Dir function

* Use walk function in folders

* Move things from ID

* Fix some tests

* Add tree

* Sync worker and changes

* Add more TODOs

* Add normal join

* Remove things in local

* Consolidate single Join

* Call it safe

* Add new IsPathSupported action

* Move the depth to resources

* Add more cases

* Improve trie implementation

* Add tests trie

* Fix trie tests

* Improve trie tests

* Add tests for walk

* Fix linting

* Add unit tests filepath

* Remove TODO

* Remove another TODO

* Unsupported file extension error

* Add documentation for IsPathSupported

* Filepath unit tests

* Use safepath to validate github path

* Remove TODO in wrapper

* Use trailing slash in folder internal object

* Fix changes test

* Include dot

* Add TODO to explore own type for path

* Fix frontend lint

* Fix unit tests

* Fix provisioning integration tests

---------

Co-authored-by: Roberto Jiménez Sánchez <roberto.jimenez@grafana.com>
Co-authored-by: Alex Khomenko <Clarity-89@users.noreply.github.com>
Co-authored-by: Ryan McKinley <ryantxu@gmail.com>
Co-authored-by: Clarity-89 <homes89@ukr.net>
Co-authored-by: Andrej Ocenas <mr.ocenas@gmail.com>
Co-authored-by: Kevin Minehart <kmineh0151@gmail.com>
This commit is contained in:
Mariell Hoversholm
2025-03-25 08:59:03 +01:00
committed by GitHub
co-authored by Ryan McKinley Alex Khomenko Clarity-89 Roberto Jimenez Sanchez Andrej Ocenas Kevin Minehart
parent e48ec7d6b6
commit 77fa2271be
269 changed files with 24541 additions and 1591 deletions
Download Patch File Download Diff File Expand all files Collapse all files
pkg/aggregator/go.mod
+12 -12
View File
@@ -4,7 +4,7 @@ go 1.23.7
require (
github.com/emicklei/go-restful/v3 v3.11.0
github.com/grafana/grafana-plugin-sdk-go v0.272.0
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0
github.com/grafana/grafana/pkg/apimachinery v0.0.0-20240808213237-f4d2e064f435
github.com/grafana/grafana/pkg/semconv v0.0.0-20240808213237-f4d2e064f435
github.com/mattbaird/jsonpatch v0.0.0-20240118010651-0ba75a80ca38
@@ -25,7 +25,7 @@ require (
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect
github.com/NYTimes/gziphandler v1.1.1 // indirect
github.com/antlr4-go/antlr/v4 v4.13.1 // indirect
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30 // indirect
github.com/apache/arrow-go/v18 v18.2.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/blang/semver/v4 v4.0.0 // indirect
@@ -49,13 +49,13 @@ require (
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/goccy/go-json v0.10.4 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v4 v4.5.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/btree v1.1.3 // indirect
github.com/google/cel-go v0.23.2 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/flatbuffers v25.2.10+incompatible // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
@@ -77,8 +77,8 @@ require (
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/jszwedko/go-datemath v0.1.1-0.20230526204004-640a500621d6 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
@@ -97,7 +97,7 @@ require (
github.com/onsi/ginkgo/v2 v2.22.0 // indirect
github.com/onsi/gomega v1.36.1 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_golang v1.21.0 // indirect
@@ -125,7 +125,7 @@ require (
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.60.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
@@ -140,16 +140,16 @@ require (
golang.org/x/net v0.36.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/term v0.29.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/tools v0.30.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/grpc v1.70.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
google.golang.org/grpc v1.71.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/fsnotify/fsnotify.v1 v1.4.7 // indirect
pkg/aggregator/go.sum
+24 -24
View File
@@ -11,8 +11,8 @@ github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7X
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/antlr4-go/antlr/v4 v4.13.1 h1:SqQKkuVZ+zWkMMNkjy5FZe5mr5WURWnlpmOuzYWrPrQ=
github.com/antlr4-go/antlr/v4 v4.13.1/go.mod h1:GKmUxMtwp6ZgGwZSva4eWPC5mS6vUAmOABFgjdkM7Nw=
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30 h1:hXVi7QKuCQ0E8Yujfu9b0f0RnzZ72efpWvPnZgnJPrE=
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30/go.mod h1:RNuWDIiGjq5nndL2PyQrndUy9nMLwheA3uWaAV7fe4U=
github.com/apache/arrow-go/v18 v18.2.0 h1:QhWqpgZMKfWOniGPhbUxrHohWnooGURqL2R2Gg4SO1Q=
github.com/apache/arrow-go/v18 v18.2.0/go.mod h1:Ic/01WSwGJWRrdAZcxjBZ5hbApNJ28K96jGYaxzzGUc=
github.com/apache/thrift v0.21.0 h1:tdPmh/ptjE1IJnhbhrcl2++TauVjy242rkV/UzJChnE=
github.com/apache/thrift v0.21.0/go.mod h1:W1H8aR/QRtYNvrPeFXBtobyRkd0/YVhTc6i07XIAgDw=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
@@ -93,8 +93,8 @@ github.com/go-task/slim-sprig/v3 v3.0.0 h1:sUs3vkvUymDpBKi3qH1YSqBQk9+9D/8M2mN1v
github.com/go-task/slim-sprig/v3 v3.0.0/go.mod h1:W848ghGpv3Qj3dhTPRyJypKRiqCdHZiAzKg9hl15HA8=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/goccy/go-json v0.10.4 h1:JSwxQzIqKfmFX1swYPpUThQZp/Ka4wzJdK0LWVytLPM=
github.com/goccy/go-json v0.10.4/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/godbus/dbus/v5 v5.0.4/go.mod h1:xhWf0FNVPg57R7Z0UbKHbJfkEywrmjJnf7w5xrFpKfA=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
@@ -113,8 +113,8 @@ github.com/google/btree v1.1.3 h1:CVpQJjYgC4VbzxeGVHfvZrv1ctoYCAI8vbl07Fcxlyg=
github.com/google/btree v1.1.3/go.mod h1:qOPhT0dTNdNzV6Z/lhRX0YXUafgPLFUh+gZMl761Gm4=
github.com/google/cel-go v0.23.2 h1:UdEe3CvQh3Nv+E/j9r1Y//WO0K0cSyD7/y0bzyLIMI4=
github.com/google/cel-go v0.23.2/go.mod h1:52Pb6QsDbC5kvgxvZhiL9QX1oZEkcUF/ZqaPx1J5Wwo=
github.com/google/flatbuffers v24.3.25+incompatible h1:CX395cjN9Kke9mmalRoL3d81AtFUxJM+yDthflgJGkI=
github.com/google/flatbuffers v24.3.25+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v25.2.10+incompatible h1:F3vclr7C3HpB1k9mxCGRMXq6FdUalZ6H/pNX4FP1v0Q=
github.com/google/flatbuffers v25.2.10+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I=
github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
@@ -136,8 +136,8 @@ github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg=
github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE=
github.com/grafana/grafana-plugin-sdk-go v0.272.0 h1:TmPIG+6e3lYGzkyfUfCHuaMaaiwDbkCacTZ7V/JaSeg=
github.com/grafana/grafana-plugin-sdk-go v0.272.0/go.mod h1:i/9KH9y/6m5hkRnG3H6aR2nOMPbJUmvo4XNrHjI15cU=
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0 h1:qVdhLR+XkVdTQ2Sr7+VnRfGM8RMp8oPe25nghsSpQms=
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0/go.mod h1:jV+CTjXqXYuaz8FgSG7ALOib3sgiDo/00dfsQFVTSpM=
github.com/grafana/grafana/pkg/apimachinery v0.0.0-20240808213237-f4d2e064f435 h1:lmw60EW7JWlAEvgggktOyVkH4hF1m/+LSF/Ap0NCyi8=
github.com/grafana/grafana/pkg/apimachinery v0.0.0-20240808213237-f4d2e064f435/go.mod h1:ORVFiW/KNRY52lNjkGwnFWCxNVfE97bJG2jr2fetq0I=
github.com/grafana/grafana/pkg/semconv v0.0.0-20240808213237-f4d2e064f435 h1:SNEeqY22DrGr5E9kGF1mKSqlOom14W9+b1u4XEGJowA=
@@ -183,10 +183,10 @@ github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -248,8 +248,8 @@ github.com/onsi/gomega v1.36.1/go.mod h1:PvZbdDc8J6XJEpDK4HCuRBm8a6Fzp9/DmhC9C7y
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s=
github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -363,8 +363,8 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 h1:sbiXRND
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0/go.mod h1:69uWxva0WgAA/4bu2Yy70SLDBwZXuQ6PbBpbsa5iZrQ=
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0 h1:D3htJISCUU/wOVlKwisVKancWm+2U4h9xDEaiMkiyRE=
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0/go.mod h1:DAX1bsj+uDm2ZuOQH/RgZRx7RQZWyzV5W2WR/0UX8JA=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0 h1:Xx1N6cDr8iWy1Cz6OcY7oS0ACdt/6HDYTdu4KskuC7s=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0/go.mod h1:iWS+NvC948FyfnJbVfPN9h/8+vr8CR2FPn6XsLRkvH8=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0 h1:VpYbyLrB5BS3blBCJMqHRIrbU4RlPnyFovR3La+1j4Q=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0/go.mod h1:XAJmM2MWhiIoTO4LCLBVeE8w009TmsYk6hq1UNdXs5A=
go.opentelemetry.io/otel v1.21.0/go.mod h1:QZzNPQPm1zLX4gZK4cMi+71eaorMSGT3A4znnUvNNEo=
go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
@@ -447,8 +447,8 @@ golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -485,18 +485,18 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 h1:Pw6WnI9W/LIdRxqK7T6XGugGbHIRl5Q7q3BssH6xk4s=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4/go.mod h1:qbZzneIOXSq+KFAFut9krLfRLZiFLzZL5u2t8SV83EE=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 h1:fCuMM4fowGzigT89NCIsW57Pk9k2D12MMi2ODn+Nk+o=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a h1:nwKuGPlUAt+aR+pcrkfFRrTU1BVrSmYyYMxYbUIVHr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a/go.mod h1:3kWAYMk1I75K4vykHtKt2ycnOgpA6974V7bREqbsenU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.18.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
pkg/apimachinery/go.mod
+3 -3
View File
@@ -40,10 +40,10 @@ require (
golang.org/x/crypto v0.35.0 // indirect
golang.org/x/net v0.36.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.22.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/grpc v1.70.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
google.golang.org/grpc v1.71.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
pkg/apimachinery/go.sum
+6 -6
View File
@@ -122,8 +122,8 @@ golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.17.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
@@ -147,10 +147,10 @@ golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8T
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
pkg/apimachinery/identity/context.go
+1 -1
View File
@@ -62,7 +62,7 @@ func WithServiceIdentity(ctx context.Context, orgID int64) (context.Context, Req
return WithRequester(ctx, r), r
}
func WithProvisioningIdentitiy(ctx context.Context, namespace string) (context.Context, Requester, error) {
func WithProvisioningIdentity(ctx context.Context, namespace string) (context.Context, Requester, error) {
ns, err := types.ParseNamespace(namespace)
if err != nil {
return nil, nil, err
pkg/apis/provisioning/v0alpha1/jobs.go
+20
View File
@@ -22,6 +22,26 @@ type JobList struct {
Items []Job `json:"items,omitempty"`
}
// HistoricJob is a history entry of Job. It is used to store Jobs that have been processed.
//
// The repository name and type are stored as labels.
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type HistoricJob struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec JobSpec `json:"spec,omitempty"`
Status JobStatus `json:"status,omitempty"`
}
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type HistoricJobList struct {
metav1.TypeMeta `json:",inline"`
metav1.ListMeta `json:"metadata,omitempty"`
Items []HistoricJob `json:"items,omitempty"`
}
// +enum
type JobAction string
pkg/apis/provisioning/v0alpha1/register.go
+31 -1
View File
@@ -68,7 +68,35 @@ var JobResourceInfo = utils.NewResourceInfo(GROUP, VERSION,
Reader: func(obj any) ([]interface{}, error) {
m, ok := obj.(*Job)
if !ok {
return nil, errors.New("expected Repository")
return nil, errors.New("expected Job")
}
return []interface{}{
m.Name, // may our may not be nice to read
m.CreationTimestamp.UTC().Format(time.RFC3339),
m.Spec.Action,
m.Status.State,
m.Status.Message,
}, nil
},
})
var HistoricJobResourceInfo = utils.NewResourceInfo(GROUP, VERSION,
"historicjobs", "historicjob", "HistoricJob",
func() runtime.Object { return &HistoricJob{} }, // newObj
func() runtime.Object { return &HistoricJobList{} }, // newList
utils.TableColumns{ // Returned by `kubectl get`. Doesn't affect disk storage.
Definition: []metav1.TableColumnDefinition{
{Name: "Name", Type: "string", Format: "name"},
{Name: "Created At", Type: "date"},
{Name: "Action", Type: "string"},
{Name: "State", Type: "string"},
{Name: "Message", Type: "string"},
},
Reader: func(obj any) ([]interface{}, error) {
m, ok := obj.(*HistoricJob)
if !ok {
return nil, errors.New("expected HistoricJob")
}
return []interface{}{
@@ -117,6 +145,8 @@ func AddKnownTypes(gv schema.GroupVersion, scheme *runtime.Scheme) error {
&ResourceStats{},
&Job{},
&JobList{},
&HistoricJob{},
&HistoricJobList{},
)
return nil
}
pkg/apis/provisioning/v0alpha1/zz_generated.deepcopy.go
+61
View File
@@ -132,6 +132,67 @@ func (in *HealthStatus) DeepCopy() *HealthStatus {
return out
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *HistoricJob) DeepCopyInto(out *HistoricJob) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ObjectMeta.DeepCopyInto(&out.ObjectMeta)
in.Spec.DeepCopyInto(&out.Spec)
in.Status.DeepCopyInto(&out.Status)
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HistoricJob.
func (in *HistoricJob) DeepCopy() *HistoricJob {
if in == nil {
return nil
}
out := new(HistoricJob)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *HistoricJob) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *HistoricJobList) DeepCopyInto(out *HistoricJobList) {
*out = *in
out.TypeMeta = in.TypeMeta
in.ListMeta.DeepCopyInto(&out.ListMeta)
if in.Items != nil {
in, out := &in.Items, &out.Items
*out = make([]HistoricJob, len(*in))
for i := range *in {
(*in)[i].DeepCopyInto(&(*out)[i])
}
}
return
}
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new HistoricJobList.
func (in *HistoricJobList) DeepCopy() *HistoricJobList {
if in == nil {
return nil
}
out := new(HistoricJobList)
in.DeepCopyInto(out)
return out
}
// DeepCopyObject is an autogenerated deepcopy function, copying the receiver, creating a new runtime.Object.
func (in *HistoricJobList) DeepCopyObject() runtime.Object {
if c := in.DeepCopy(); c != nil {
return c
}
return nil
}
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *HistoryItem) DeepCopyInto(out *HistoryItem) {
*out = *in
pkg/apis/provisioning/v0alpha1/zz_generated.openapi.go
+96
View File
@@ -20,6 +20,8 @@ func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenA
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.FileList": schema_pkg_apis_provisioning_v0alpha1_FileList(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.GitHubRepositoryConfig": schema_pkg_apis_provisioning_v0alpha1_GitHubRepositoryConfig(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HealthStatus": schema_pkg_apis_provisioning_v0alpha1_HealthStatus(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoricJob": schema_pkg_apis_provisioning_v0alpha1_HistoricJob(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoricJobList": schema_pkg_apis_provisioning_v0alpha1_HistoricJobList(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoryItem": schema_pkg_apis_provisioning_v0alpha1_HistoryItem(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoryList": schema_pkg_apis_provisioning_v0alpha1_HistoryList(ref),
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.Job": schema_pkg_apis_provisioning_v0alpha1_Job(ref),
@@ -336,6 +338,100 @@ func schema_pkg_apis_provisioning_v0alpha1_HealthStatus(ref common.ReferenceCall
}
}
func schema_pkg_apis_provisioning_v0alpha1_HistoricJob(ref common.ReferenceCallback) common.OpenAPIDefinition {
return common.OpenAPIDefinition{
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "HistoricJob is a history entry of Job. It is used to store Jobs that have been processed.\n\nThe repository name and type are stored as labels.",
Type: []string{"object"},
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Default: map[string]interface{}{},
Ref: ref("k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta"),
},
},
"spec": {
SchemaProps: spec.SchemaProps{
Default: map[string]interface{}{},
Ref: ref("github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.JobSpec"),
},
},
"status": {
SchemaProps: spec.SchemaProps{
Default: map[string]interface{}{},
Ref: ref("github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.JobStatus"),
},
},
},
},
},
Dependencies: []string{
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.JobSpec", "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.JobStatus", "k8s.io/apimachinery/pkg/apis/meta/v1.ObjectMeta"},
}
}
func schema_pkg_apis_provisioning_v0alpha1_HistoricJobList(ref common.ReferenceCallback) common.OpenAPIDefinition {
return common.OpenAPIDefinition{
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Type: []string{"object"},
Properties: map[string]spec.Schema{
"kind": {
SchemaProps: spec.SchemaProps{
Description: "Kind is a string value representing the REST resource this object represents. Servers may infer this from the endpoint the client submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds",
Type: []string{"string"},
Format: "",
},
},
"apiVersion": {
SchemaProps: spec.SchemaProps{
Description: "APIVersion defines the versioned schema of this representation of an object. Servers should convert recognized schemas to the latest internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources",
Type: []string{"string"},
Format: "",
},
},
"metadata": {
SchemaProps: spec.SchemaProps{
Default: map[string]interface{}{},
Ref: ref("k8s.io/apimachinery/pkg/apis/meta/v1.ListMeta"),
},
},
"items": {
SchemaProps: spec.SchemaProps{
Type: []string{"array"},
Items: &spec.SchemaOrArray{
Schema: &spec.Schema{
SchemaProps: spec.SchemaProps{
Default: map[string]interface{}{},
Ref: ref("github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoricJob"),
},
},
},
},
},
},
},
},
Dependencies: []string{
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1.HistoricJob", "k8s.io/apimachinery/pkg/apis/meta/v1.ListMeta"},
}
}
func schema_pkg_apis_provisioning_v0alpha1_HistoryItem(ref common.ReferenceCallback) common.OpenAPIDefinition {
return common.OpenAPIDefinition{
Schema: spec.Schema{
pkg/apis/secret/go.mod
+5 -5
View File
@@ -5,7 +5,7 @@ go 1.23.7
require (
github.com/grafana/grafana/pkg/apimachinery v0.0.0-20250314071911-14e2784e6979
github.com/stretchr/testify v1.10.0
google.golang.org/grpc v1.70.0
google.golang.org/grpc v1.71.0
google.golang.org/protobuf v1.36.5
gopkg.in/yaml.v3 v3.0.1
k8s.io/apimachinery v0.32.1
@@ -45,7 +45,7 @@ require (
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@@ -79,14 +79,14 @@ require (
go.uber.org/zap v1.27.0 // indirect
golang.org/x/net v0.36.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/term v0.29.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/tools v0.30.0 // indirect
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
k8s.io/api v0.32.1 // indirect
pkg/apis/secret/go.sum
+10 -10
View File
@@ -99,8 +99,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -259,8 +259,8 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -292,18 +292,18 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 h1:Pw6WnI9W/LIdRxqK7T6XGugGbHIRl5Q7q3BssH6xk4s=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4/go.mod h1:qbZzneIOXSq+KFAFut9krLfRLZiFLzZL5u2t8SV83EE=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 h1:fCuMM4fowGzigT89NCIsW57Pk9k2D12MMi2ODn+Nk+o=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a h1:nwKuGPlUAt+aR+pcrkfFRrTU1BVrSmYyYMxYbUIVHr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a/go.mod h1:3kWAYMk1I75K4vykHtKt2ycnOgpA6974V7bREqbsenU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.18.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
pkg/apiserver/go.mod
+6 -6
View File
@@ -1,6 +1,6 @@
module github.com/grafana/grafana/pkg/apiserver
go 1.23.7
go 1.24.1
require (
github.com/google/go-cmp v0.7.0
@@ -51,7 +51,7 @@ require (
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
@@ -83,15 +83,15 @@ require (
go.uber.org/zap v1.27.0 // indirect
golang.org/x/net v0.36.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/term v0.29.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/time v0.9.0 // indirect
golang.org/x/tools v0.30.0 // indirect
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/grpc v1.70.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
google.golang.org/grpc v1.71.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
pkg/apiserver/go.sum
+10 -10
View File
@@ -103,8 +103,8 @@ github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnr
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI2bnpBCr8=
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
@@ -267,8 +267,8 @@ golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7w
golang.org/x/sys v0.0.0-20190422165155-953cdadca894/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.29.0 h1:L6pJp37ocefwRRtYPKSWOWzOtWSxVajvz2ldH/xi3iU=
golang.org/x/term v0.29.0/go.mod h1:6bl4lRlvVuDgSf3179VpIxBF0o10JUpXWOnI7nErv7s=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -300,18 +300,18 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20200423170343-7949de9c1215/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 h1:Pw6WnI9W/LIdRxqK7T6XGugGbHIRl5Q7q3BssH6xk4s=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4/go.mod h1:qbZzneIOXSq+KFAFut9krLfRLZiFLzZL5u2t8SV83EE=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 h1:fCuMM4fowGzigT89NCIsW57Pk9k2D12MMi2ODn+Nk+o=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a h1:nwKuGPlUAt+aR+pcrkfFRrTU1BVrSmYyYMxYbUIVHr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a/go.mod h1:3kWAYMk1I75K4vykHtKt2ycnOgpA6974V7bREqbsenU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.18.0/go.mod h1:6QZJwpn2B+Zp71q/5VxRsJ6NXXVCE5NRUHRo+f3cWCs=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
pkg/apiserver/readonly/store.go
+160
View File
@@ -0,0 +1,160 @@
package readonly
import (
"context"
"errors"
"net/http"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metainternalversion "k8s.io/apimachinery/pkg/apis/meta/internalversion"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/watch"
"k8s.io/apiserver/pkg/registry/rest"
)
var (
// ErrOperationUnsupported is returned when a read operation is not supported by the underlying storage.
// If you wish to check for this, call IsOperationUnsupported instead.
//
// Some methods may even include the name of the object that caused the error.
ErrOperationUnsupported = &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "Operation not supported",
Code: http.StatusMethodNotAllowed,
Status: metav1.StatusFailure,
Reason: metav1.StatusReasonMethodNotAllowed,
Details: &metav1.StatusDetails{
Causes: []metav1.StatusCause{
{
Type: "OperationNotImplementedByInner",
Message: "Operation not supported by the wrapped storage",
},
},
},
},
}
_ rest.Storage = readOnly{}
_ rest.Scoper = readOnly{}
_ rest.SingularNameProvider = readOnly{}
_ rest.StorageWithReadiness = readOnly{}
_ rest.Getter = readOnly{}
_ rest.Lister = readOnlyLister{}
_ rest.TableConvertor = readOnlyLister{} // due to rest.Lister
_ rest.Watcher = readOnly{}
)
// ReadOnly wraps a storage interface and makes all methods read-only.
//
// One write operation is passed through: Destroy. This allows you to not have to have some complex clean up setup.
type readOnly struct {
inner rest.Storage
}
// ReadOnlyLister is like ReadOnly, but also implements rest.Lister.
type readOnlyLister struct {
readOnly
inner rest.Lister
}
func Wrap(store rest.Storage) rest.Storage {
ro := readOnly{inner: store}
if l, ok := store.(rest.Lister); ok {
return readOnlyLister{readOnly: ro, inner: l}
}
return ro
}
func (ro readOnly) New() runtime.Object {
return ro.inner.New()
}
func (ro readOnlyLister) NewList() runtime.Object {
return ro.inner.NewList()
}
func (ro readOnly) Destroy() {
ro.inner.Destroy()
}
func (ro readOnly) NamespaceScoped() bool {
if s, ok := ro.inner.(rest.Scoper); ok {
return s.NamespaceScoped()
}
return false
}
func (ro readOnly) GetSingularName() string {
if s, ok := ro.inner.(rest.SingularNameProvider); ok {
return s.GetSingularName()
}
return ""
}
// ReadinessCheck does not check for readiness if the inner storage does not implement it.
// In that case, the inner storage is always considered ready.
func (ro readOnly) ReadinessCheck() error {
if r, ok := ro.inner.(rest.StorageWithReadiness); ok {
return r.ReadinessCheck()
}
return nil
}
func (ro readOnly) Get(ctx context.Context, name string, options *metav1.GetOptions) (runtime.Object, error) {
if r, ok := ro.inner.(rest.Getter); ok {
return r.Get(ctx, name, options)
}
return nil, ro.unsupported(name)
}
func (ro readOnlyLister) List(ctx context.Context, options *metainternalversion.ListOptions) (runtime.Object, error) {
return ro.inner.List(ctx, options)
}
func (ro readOnlyLister) ConvertToTable(ctx context.Context, object runtime.Object, tableOptions runtime.Object) (*metav1.Table, error) {
return ro.inner.ConvertToTable(ctx, object, tableOptions)
}
func (ro readOnly) Watch(ctx context.Context, options *metainternalversion.ListOptions) (watch.Interface, error) {
if r, ok := ro.inner.(rest.Watcher); ok {
return r.Watch(ctx, options)
}
return nil, ro.unsupported("")
}
func (ro readOnly) unsupported(name string) error {
obj := ro.New()
gvk := obj.GetObjectKind().GroupVersionKind()
err := *ErrOperationUnsupported
err.ErrStatus.Details.Kind = gvk.Kind
err.ErrStatus.Details.Group = gvk.Group
err.ErrStatus.Details.Name = name
return &err
}
// IsOperationUnsupported checks the error for its reasoning. If it originated from the read-only wrapper, it will return true.
// When this returns true, it indicates the underlying storage does not support the operation despite us "announcing" it (by implementing an interface).
func IsOperationUnsupported(err error) bool {
var statusErr *apierrors.StatusError
if !errors.As(err, &statusErr) {
return false
}
if statusErr.Status().Reason != metav1.StatusReasonMethodNotAllowed ||
statusErr.Status().Details == nil ||
len(statusErr.Status().Details.Causes) == 0 {
return false
}
for _, detail := range statusErr.Status().Details.Causes {
if detail.Type == "OperationNotImplementedByInner" {
return true
}
}
return false
}
pkg/apiserver/registry/generic/storage.go
+15 -1
View File
@@ -9,7 +9,9 @@ import (
)
func NewRegistryStore(scheme *runtime.Scheme, resourceInfo utils.ResourceInfo, optsGetter generic.RESTOptionsGetter) (*registry.Store, error) {
strategy := NewStrategy(scheme, resourceInfo.GroupVersion())
gv := resourceInfo.GroupVersion()
gv.Version = runtime.APIVersionInternal
strategy := NewStrategy(scheme, gv)
store := &registry.Store{
NewFunc: resourceInfo.NewFunc,
NewListFunc: resourceInfo.NewListFunc,
@@ -30,6 +32,18 @@ func NewRegistryStore(scheme *runtime.Scheme, resourceInfo utils.ResourceInfo, o
return store, nil
}
func NewCompleteRegistryStore(scheme *runtime.Scheme, resourceInfo utils.ResourceInfo, optsGetter generic.RESTOptionsGetter) (*registry.Store, error) {
registryStore, err := NewRegistryStore(scheme, resourceInfo, optsGetter)
if err != nil {
return nil, err
}
strategy := NewCompleteStrategy(scheme, resourceInfo.GroupVersion())
registryStore.CreateStrategy = strategy
registryStore.UpdateStrategy = strategy
registryStore.DeleteStrategy = strategy
return registryStore, nil
}
func NewRegistryStatusStore(scheme *runtime.Scheme, specStore *registry.Store) *StatusREST {
gv := specStore.New().GetObjectKind().GroupVersionKind().GroupVersion()
strategy := NewStatusStrategy(scheme, gv)
pkg/apiserver/registry/generic/strategy.go
+69
View File
@@ -16,6 +16,8 @@ import (
"github.com/grafana/grafana/pkg/apimachinery/utils"
)
// genericStrategy allows for writing objects with spec fields.
// It ignores status fields, and does not allow for status updates.
type genericStrategy struct {
runtime.ObjectTyper
names.NameGenerator
@@ -111,6 +113,8 @@ func (g *genericStrategy) WarningsOnUpdate(ctx context.Context, obj, old runtime
return nil
}
// genericStatusStrategy allows for writing objects with status fields, however may not create them.
// It ignores spec and metadata fields, and does not allow for updates outside of the status field.
type genericStatusStrategy struct {
runtime.ObjectTyper
names.NameGenerator
@@ -176,6 +180,71 @@ func (g *genericStatusStrategy) WarningsOnUpdate(ctx context.Context, obj, old r
return nil
}
// genericCompleteStrategy allows for writing objects with spec and status fields.
// It does not ignore any fields, and allows for updates to both spec and status fields.
// This is the same as having separate stores for spec and status fields.
//
// This can be applied to both the root object and status subresource in a Kubernetes REST API.
type genericCompleteStrategy struct {
runtime.ObjectTyper
names.NameGenerator
gv schema.GroupVersion
}
// NewCompleteStrategy creates a new genericCompleteStrategy.
func NewCompleteStrategy(typer runtime.ObjectTyper, gv schema.GroupVersion) *genericCompleteStrategy {
return &genericCompleteStrategy{typer, names.SimpleNameGenerator, gv}
}
func (g *genericCompleteStrategy) NamespaceScoped() bool {
return true
}
func (g *genericCompleteStrategy) GetResetFields() map[fieldpath.APIVersion]*fieldpath.Set {
fields := map[fieldpath.APIVersion]*fieldpath.Set{
fieldpath.APIVersion(g.gv.String()): fieldpath.NewSet(),
}
return fields
}
func (g *genericCompleteStrategy) PrepareForCreate(ctx context.Context, obj runtime.Object) {
meta, err := utils.MetaAccessor(obj)
if err != nil {
return
}
meta.SetGeneration(1)
}
func (g *genericCompleteStrategy) PrepareForUpdate(ctx context.Context, obj, old runtime.Object) {}
func (g *genericCompleteStrategy) AllowCreateOnUpdate() bool {
return true
}
func (g *genericCompleteStrategy) AllowUnconditionalUpdate() bool {
return true
}
func (g *genericCompleteStrategy) Canonicalize(obj runtime.Object) {}
func (g *genericCompleteStrategy) Validate(ctx context.Context, obj runtime.Object) field.ErrorList {
return nil
}
func (g *genericCompleteStrategy) ValidateUpdate(ctx context.Context, obj, old runtime.Object) field.ErrorList {
return nil
}
func (g *genericCompleteStrategy) WarningsOnCreate(ctx context.Context, obj runtime.Object) []string {
return nil
}
func (g *genericCompleteStrategy) WarningsOnUpdate(ctx context.Context, obj, old runtime.Object) []string {
return nil
}
// GetAttrs returns labels and fields of an object.
func GetAttrs(obj runtime.Object) (labels.Set, fields.Set, error) {
accessor, err := meta.Accessor(obj)
pkg/apiserver/registry/generic/strategy_test.go
+440 -79
View File
@@ -1,104 +1,465 @@
package generic_test
import (
"context"
"testing"
"github.com/grafana/grafana/pkg/apiserver/registry/generic"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/fields"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apiserver/pkg/apis/example"
"github.com/grafana/grafana/pkg/apiserver/registry/generic"
)
func TestPrepareForUpdate(t *testing.T) {
ctx := context.TODO()
func TestGenericStrategy(t *testing.T) {
t.Parallel()
gv := schema.GroupVersion{Group: "test", Version: "v1"}
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
oldObj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("PrepareForUpdate", func(t *testing.T) {
t.Parallel()
testCases := []struct {
name string
newObj *example.Pod
oldObj *example.Pod
expectedGen int64
expectedObj *example.Pod
}{
{
name: "ignore status updates",
newObj: &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Stopped"),
},
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
oldObj: oldObj.DeepCopy(),
expectedGen: 2,
expectedObj: &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
},
}
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
strategy.PrepareForUpdate(ctx, tc.newObj, tc.oldObj)
require.Equal(t, tc.expectedObj, tc.newObj)
t.Run("ignores status updates", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Status.Phase = example.PodPhase("Stopped")
expectedObj := obj.DeepCopy()
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
}
t.Run("does not increment generation if annotations, labels, finalizers, or owner references change", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Annotations = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Labels = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Finalizers = []string{"foo"}
newObj.ObjectMeta.OwnerReferences = []metav1.OwnerReference{{Name: "foo"}}
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
assert.Equal(t, map[string]string{"foo": "baz"}, newObj.ObjectMeta.Annotations)
assert.Equal(t, map[string]string{"foo": "baz"}, newObj.ObjectMeta.Labels)
assert.Equal(t, []string{"foo"}, newObj.ObjectMeta.Finalizers)
assert.Equal(t, []metav1.OwnerReference{{Name: "foo"}}, newObj.ObjectMeta.OwnerReferences)
assert.Equal(t, int64(1), newObj.Generation)
})
t.Run("does not increment generation if spec changes", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Spec.NodeSelector = map[string]string{"foo": "baz"}
expectedObj := newObj.DeepCopy()
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
})
t.Run("PrepareForCreate", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("assigns generation=1", func(t *testing.T) {
t.Parallel()
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
obj := obj.DeepCopy()
strategy.PrepareForCreate(t.Context(), obj)
require.Equal(t, int64(1), obj.Generation)
})
t.Run("clears status", func(t *testing.T) {
t.Parallel()
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
obj := obj.DeepCopy()
strategy.PrepareForCreate(t.Context(), obj)
require.Equal(t, example.PodStatus{}, obj.Status)
})
t.Run("leaves spec untouched", func(t *testing.T) {
t.Parallel()
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
obj := obj.DeepCopy()
originalSpec := *obj.Spec.DeepCopy()
strategy.PrepareForCreate(t.Context(), obj)
require.Equal(t, originalSpec, obj.Spec)
})
})
}
func TestPrepareForCreate(t *testing.T) {
ctx := context.TODO()
func TestStatusStrategy(t *testing.T) {
t.Parallel()
gv := schema.GroupVersion{Group: "test", Version: "v1"}
strategy := generic.NewStrategy(runtime.NewScheme(), gv)
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("PrepareForUpdate", func(t *testing.T) {
t.Parallel()
strategy.PrepareForCreate(ctx, obj)
require.Equal(t, int64(1), obj.Generation)
require.Equal(t, example.PodStatus{}, obj.Status)
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("ignores spec updates", func(t *testing.T) {
// The assumption here is that the status strategy should not allow for spec updates.
// This is drawn due to the GetResetFields function returning `metadata` and `spec`, and due to it copying old `metadata` fields to the new object (but not spec?).
t.Skip("assumption does not hold -- verify with app platform if this is intended")
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Spec.NodeSelector = map[string]string{"foo": "baz"}
expectedObj := obj.DeepCopy()
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
t.Run("ignores label updates", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Labels = map[string]string{"foo": "baz"}
expectedObj := obj.DeepCopy()
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
t.Run("ignores annotation updates", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Annotations = map[string]string{"foo": "baz"}
expectedObj := obj.DeepCopy()
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
t.Run("ignores finalizer updates", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Finalizers = []string{"foo"}
expectedObj := obj.DeepCopy()
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
t.Run("ignores owner references updates", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.OwnerReferences = []metav1.OwnerReference{{Name: "foo"}}
expectedObj := obj.DeepCopy()
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, expectedObj, newObj)
})
t.Run("does not increment generation on status changes", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Status.Phase = example.PodPhase("Stopped")
strategy := generic.NewStatusStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, int64(1), newObj.Generation)
})
})
}
func TestCompleteStrategy(t *testing.T) {
t.Parallel()
gv := schema.GroupVersion{Group: "test", Version: "v1"}
t.Run("PrepareForUpdate", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Generation: 1,
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("on status updates", func(t *testing.T) {
t.Parallel()
t.Run("keeps the change", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Status.Phase = example.PodPhase("Stopped")
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, example.PodPhase("Stopped"), newObj.Status.Phase)
})
t.Run("does not change generation", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Status.Phase = example.PodPhase("Stopped")
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, int64(1), newObj.Generation)
})
})
t.Run("on spec updates", func(t *testing.T) {
t.Parallel()
t.Run("keeps the change", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Spec.NodeSelector = map[string]string{"foo": "baz"}
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, map[string]string{"foo": "baz"}, newObj.Spec.NodeSelector)
})
t.Run("does not increment generation", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.Spec.NodeSelector = map[string]string{"foo": "baz"}
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
require.Equal(t, int64(1), newObj.Generation)
})
})
t.Run("on metadata updates", func(t *testing.T) {
t.Parallel()
t.Run("keeps the change", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Annotations = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Labels = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Finalizers = []string{"foo"}
newObj.ObjectMeta.OwnerReferences = []metav1.OwnerReference{{Name: "foo"}}
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
assert.Equal(t, map[string]string{"foo": "baz"}, newObj.ObjectMeta.Annotations)
assert.Equal(t, map[string]string{"foo": "baz"}, newObj.ObjectMeta.Labels)
assert.Equal(t, []string{"foo"}, newObj.ObjectMeta.Finalizers)
assert.Equal(t, []metav1.OwnerReference{{Name: "foo"}}, newObj.ObjectMeta.OwnerReferences)
})
t.Run("does not increment generation", func(t *testing.T) {
t.Parallel()
oldObj := obj.DeepCopy()
newObj := obj.DeepCopy()
newObj.ObjectMeta.Annotations = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Labels = map[string]string{"foo": "baz"}
newObj.ObjectMeta.Finalizers = []string{"foo"}
newObj.ObjectMeta.OwnerReferences = []metav1.OwnerReference{{Name: "foo"}}
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
strategy.PrepareForUpdate(t.Context(), newObj, oldObj)
assert.Equal(t, int64(1), newObj.Generation)
})
})
})
t.Run("PrepareForCreate", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
},
Spec: example.PodSpec{
NodeSelector: map[string]string{"foo": "bar"},
},
Status: example.PodStatus{
Phase: example.PodPhase("Running"),
},
}
t.Run("assigns generation=1", func(t *testing.T) {
t.Parallel()
t.Run("when generation is not set", func(t *testing.T) {
t.Parallel()
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
obj := obj.DeepCopy()
strategy.PrepareForCreate(t.Context(), obj)
require.Equal(t, int64(1), obj.Generation)
})
t.Run("when generation is set to a higher value", func(t *testing.T) {
t.Parallel()
strategy := generic.NewCompleteStrategy(runtime.NewScheme(), gv)
obj := obj.DeepCopy()
obj.Generation = 2
strategy.PrepareForCreate(t.Context(), obj)
require.Equal(t, int64(1), obj.Generation)
})
})
})
}
func TestGetAttrs(t *testing.T) {
t.Parallel()
t.Run("returns all labels", func(t *testing.T) {
t.Parallel()
t.Run("when labels is nil", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Labels: nil,
},
}
labels, _, err := generic.GetAttrs(obj)
require.NoError(t, err)
require.Empty(t, labels, "expected no labels")
})
t.Run("when there are no labels", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Labels: make(map[string]string),
},
}
labels, _, err := generic.GetAttrs(obj)
require.NoError(t, err)
require.Empty(t, labels, "expected no labels")
})
t.Run("when there is only 1 label", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Labels: map[string]string{"foo": "bar"},
},
}
l, _, err := generic.GetAttrs(obj)
require.NoError(t, err)
require.Equal(t, labels.Set{"foo": "bar"}, l, "expected labels to match")
})
t.Run("when there are many labels", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
Labels: map[string]string{"foo": "bar", "baz": "qux", "grafana": "is-cool"},
},
}
l, _, err := generic.GetAttrs(obj)
require.NoError(t, err)
require.Equal(t, labels.Set{"foo": "bar", "baz": "qux", "grafana": "is-cool"}, l, "expected labels to match")
})
})
t.Run("includes only name in fields", func(t *testing.T) {
t.Parallel()
obj := &example.Pod{
ObjectMeta: metav1.ObjectMeta{
Name: "test",
Namespace: "default",
},
}
_, f, err := generic.GetAttrs(obj)
require.NoError(t, err)
require.Equal(t, fields.Set{"metadata.name": "test"}, f, "expected fields to match")
})
}
pkg/build/go.mod
+7 -7
View File
@@ -34,7 +34,7 @@ require (
golang.org/x/text v0.22.0 // indirect; @grafana/grafana-backend-group
golang.org/x/time v0.9.0 // indirect; @grafana/grafana-backend-group
google.golang.org/api v0.220.0 // @grafana/grafana-backend-group
google.golang.org/grpc v1.70.0 // indirect; @grafana/plugins-platform-backend
google.golang.org/grpc v1.71.0 // indirect; @grafana/plugins-platform-backend
google.golang.org/protobuf v1.36.5 // indirect; @grafana/plugins-platform-backend
gopkg.in/yaml.v3 v3.0.1 // @grafana/alerting-backend
)
@@ -72,10 +72,10 @@ require (
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
go.starlark.net v0.0.0-20230525235612-a134d8f9ddca // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 // indirect; @grafana/grafana-backend-group
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
gopkg.in/yaml.v2 v2.4.0 // indirect
)
@@ -93,10 +93,10 @@ require (
github.com/adrg/xdg v0.4.0 // indirect
github.com/cenkalti/backoff/v4 v4.3.0 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78 // indirect
github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/distribution/reference v0.6.0 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.32.3 // indirect
github.com/envoyproxy/go-control-plane/envoy v1.32.4 // indirect
github.com/envoyproxy/protoc-gen-validate v1.2.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.1 // indirect
github.com/mitchellh/go-homedir v1.1.0 // indirect
@@ -106,7 +106,7 @@ require (
github.com/sosodev/duration v1.2.0 // indirect
github.com/vektah/gqlparser/v2 v2.5.20 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.33.0 // indirect
go.opentelemetry.io/contrib/detectors/gcp v1.34.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.0.0-20240518090000-14441aefdf88 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.4.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 // indirect
pkg/build/go.sum
+14 -14
View File
@@ -70,8 +70,8 @@ github.com/chzyer/readline v0.0.0-20180603132655-2972be24d48e/go.mod h1:nSuG5e5P
github.com/chzyer/test v0.0.0-20180213035817-a1ea475d72b1/go.mod h1:Q3SI9o4m/ZMnBNeIyt5eFwwo7qiLfzFZmjNmxjkiQlU=
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc=
github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78 h1:QVw89YDxXxEe+l8gU8ETbOasdwEV+avkR75ZzsVV9WI=
github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3 h1:boJj011Hh+874zpIySeApCX4GeOjPl9qhRF3QuIZq+Q=
github.com/cncf/xds/go v0.0.0-20241223141626-cff3c89139a3/go.mod h1:W+zGtBO5Y1IgJhy4+A9GOqVhqLpfZi+vwmdNXUehLA8=
github.com/containerd/containerd v1.3.4/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA=
github.com/containerd/log v0.1.0 h1:TCJt7ioM2cr/tfR8GPbGf9/VRAX8D2B4PjzCpfX540I=
github.com/containerd/log v0.1.0/go.mod h1:VRRf09a7mHDIRezVKTRCrOq78v577GXq3bSa3EhrzVo=
@@ -110,8 +110,8 @@ github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.m
github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98=
github.com/envoyproxy/go-control-plane v0.13.4 h1:zEqyPVyku6IvWCFwux4x9RxkLOMUL+1vC9xUFv5l2/M=
github.com/envoyproxy/go-control-plane v0.13.4/go.mod h1:kDfuBlDVsSj2MjrLEtRWtHlsWIFcGyB2RMO44Dc5GZA=
github.com/envoyproxy/go-control-plane/envoy v1.32.3 h1:hVEaommgvzTjTd4xCaFd+kEQ2iYBtGxP6luyLrx6uOk=
github.com/envoyproxy/go-control-plane/envoy v1.32.3/go.mod h1:F6hWupPfh75TBXGKA++MCT/CZHFq5r9/uwt/kQYkZfE=
github.com/envoyproxy/go-control-plane/envoy v1.32.4 h1:jb83lalDRZSpPWW2Z7Mck/8kXZ5CQAFYVjQcdVIr83A=
github.com/envoyproxy/go-control-plane/envoy v1.32.4/go.mod h1:Gzjc5k8JcJswLjAx1Zm+wSYE20UrLtt7JZMWiWQXQEw=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0 h1:/G9QYbddjL25KvtKTv3an9lx6VBE2cnb8wp1vEGNYGI=
github.com/envoyproxy/go-control-plane/ratelimit v0.1.0/go.mod h1:Wk+tMFAFbCXaJPzVVHnPgRKdUdwW/KdbRt94AzgRee4=
github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
@@ -251,8 +251,8 @@ github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9de
github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/contrib/detectors/gcp v1.33.0 h1:FVPoXEoILwgbZUu4X7YSgsESsAmGRgoYcnXkzgQPhP4=
go.opentelemetry.io/contrib/detectors/gcp v1.33.0/go.mod h1:ZHrLmr4ikK2AwRj9QL+c9s2SOlgoSRyMpNVzUj2fZqI=
go.opentelemetry.io/contrib/detectors/gcp v1.34.0 h1:JRxssobiPg23otYU5SbWtQC//snGVIM3Tx6QRzlQBao=
go.opentelemetry.io/contrib/detectors/gcp v1.34.0/go.mod h1:cV4BMFcscUR/ckqLkbfQmF0PRsq8w/lMGzdbCSveBHo=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 h1:rgMkmiGfix9vFJDcDi1PK8WEQP4FLQwLDfhp5ZLpFeE=
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0/go.mod h1:ijPqXp5P6IRRByFVVg9DY8P5HkxkHE5ARIa+86aXPf4=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 h1:sbiXRNDSWJOTobXh5HyQKjq6wUC5tNybqjIqDpAY4CU=
@@ -335,8 +335,8 @@ golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
@@ -366,17 +366,17 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98
google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4 h1:Pw6WnI9W/LIdRxqK7T6XGugGbHIRl5Q7q3BssH6xk4s=
google.golang.org/genproto v0.0.0-20250122153221-138b5a5a4fd4/go.mod h1:qbZzneIOXSq+KFAFut9krLfRLZiFLzZL5u2t8SV83EE=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 h1:fCuMM4fowGzigT89NCIsW57Pk9k2D12MMi2ODn+Nk+o=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a h1:nwKuGPlUAt+aR+pcrkfFRrTU1BVrSmYyYMxYbUIVHr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a/go.mod h1:3kWAYMk1I75K4vykHtKt2ycnOgpA6974V7bREqbsenU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY=
google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
google.golang.org/grpc v1.29.1/go.mod h1:itym6AZVZYACWQqET3MqgPpjcuV5QH3BxFS3IjizoKk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
pkg/plugins/codegen/go.mod
+1
View File
@@ -46,6 +46,7 @@ require (
golang.org/x/net v0.36.0 // indirect
golang.org/x/oauth2 v0.24.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/tools v0.30.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
pkg/plugins/codegen/go.sum
+2 -2
View File
@@ -100,8 +100,8 @@ golang.org/x/oauth2 v0.24.0 h1:KTBBxWqUa0ykRPLtV69rRto9TLXcqYkeswu48x/gvNE=
golang.org/x/oauth2 v0.24.0/go.mod h1:XYTD2NtWslqkgxebSiOHnXEap4TF09sJSc7H1sXbhtI=
golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
golang.org/x/tools v0.30.0 h1:BgcpHewrV5AUp2G9MebG4XPFI1E2W41zU1SaqVA9vJY=
pkg/promlib/go.mod
+12 -12
View File
@@ -4,7 +4,7 @@ go 1.23.7
require (
github.com/grafana/dskit v0.0.0-20241105154643-a6b453a88040
github.com/grafana/grafana-plugin-sdk-go v0.272.0
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0
github.com/json-iterator/go v1.1.12
github.com/prometheus/client_golang v1.21.0
github.com/prometheus/common v0.62.0
@@ -19,7 +19,7 @@ require (
github.com/Azure/azure-sdk-for-go/sdk/azcore v1.17.0 // indirect
github.com/Azure/azure-sdk-for-go/sdk/azidentity v1.8.1 // indirect
github.com/BurntSushi/toml v1.4.1-0.20240526193622-a339e1f7089c // indirect
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30 // indirect
github.com/apache/arrow-go/v18 v18.2.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go v1.55.6 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect
@@ -42,11 +42,11 @@ require (
github.com/go-openapi/jsonpointer v0.21.0 // indirect
github.com/go-openapi/jsonreference v0.21.0 // indirect
github.com/go-openapi/swag v0.23.0 // indirect
github.com/goccy/go-json v0.10.4 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/gogo/protobuf v1.3.2 // indirect
github.com/golang-jwt/jwt/v5 v5.2.2 // indirect
github.com/golang/protobuf v1.5.4 // indirect
github.com/google/flatbuffers v24.3.25+incompatible // indirect
github.com/google/flatbuffers v25.2.10+incompatible // indirect
github.com/google/gnostic-models v0.6.8 // indirect
github.com/google/go-cmp v0.7.0 // indirect
github.com/google/gofuzz v1.2.0 // indirect
@@ -64,8 +64,8 @@ require (
github.com/invopop/jsonschema v0.13.0 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jszwedko/go-datemath v0.1.1-0.20230526204004-640a500621d6 // indirect
github.com/klauspost/compress v1.17.11 // indirect
github.com/klauspost/cpuid/v2 v2.2.9 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.10 // indirect
github.com/magefile/mage v1.15.0 // indirect
github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattetti/filebuffer v1.0.1 // indirect
@@ -81,7 +81,7 @@ require (
github.com/oklog/run v1.1.0 // indirect
github.com/olekukonko/tablewriter v0.0.5 // indirect
github.com/perimeterx/marshmallow v1.1.5 // indirect
github.com/pierrec/lz4/v4 v4.1.21 // indirect
github.com/pierrec/lz4/v4 v4.1.22 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
github.com/prometheus/procfs v0.15.1 // indirect
@@ -101,7 +101,7 @@ require (
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.59.0 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/httptrace/otelhttptrace v0.60.0 // indirect
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0 // indirect
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.34.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.34.0 // indirect
go.opentelemetry.io/otel/metric v1.35.0 // indirect
@@ -113,14 +113,14 @@ require (
golang.org/x/mod v0.23.0 // indirect
golang.org/x/net v0.36.0 // indirect
golang.org/x/sync v0.11.0 // indirect
golang.org/x/sys v0.30.0 // indirect
golang.org/x/sys v0.31.0 // indirect
golang.org/x/text v0.22.0 // indirect
golang.org/x/tools v0.30.0 // indirect
golang.org/x/xerrors v0.0.0-20240903120638-7835f813f4da // indirect
google.golang.org/api v0.220.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 // indirect
google.golang.org/grpc v1.70.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b // indirect
google.golang.org/grpc v1.71.0 // indirect
google.golang.org/protobuf v1.36.5 // indirect
gopkg.in/fsnotify/fsnotify.v1 v1.4.7 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
pkg/promlib/go.sum
+25 -24
View File
@@ -1,3 +1,4 @@
cloud.google.com/go v0.118.0 h1:tvZe1mgqRxpiVa3XlIGMiPcEUbP1gNXELgD4y/IXmeQ=
cloud.google.com/go/auth v0.14.1 h1:AwoJbzUdxA/whv1qj3TLKwh3XX5sikny2fc40wUl+h0=
cloud.google.com/go/auth v0.14.1/go.mod h1:4JHUxlGXisL0AW8kXPtUF6ztuOksyfUQNFjfsOCXkPM=
cloud.google.com/go/auth/oauth2adapt v0.2.7 h1:/Lc7xODdqcEw8IrZ9SvwnlLX6j9FHQM74z6cBk9Rw6M=
@@ -20,8 +21,8 @@ github.com/alecthomas/units v0.0.0-20240927000941-0f3dac36c52b h1:mimo19zliBX/vS
github.com/alecthomas/units v0.0.0-20240927000941-0f3dac36c52b/go.mod h1:fvzegU4vN3H1qMT+8wDmzjAcDONcgo2/SZ/TyfdUOFs=
github.com/andybalholm/brotli v1.1.1 h1:PR2pgnyFznKEugtsUo0xLdDop5SKXd5Qf5ysW+7XdTA=
github.com/andybalholm/brotli v1.1.1/go.mod h1:05ib4cKhjx3OQYUY22hTVd34Bc8upXjOLL2rKwwZBoA=
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30 h1:hXVi7QKuCQ0E8Yujfu9b0f0RnzZ72efpWvPnZgnJPrE=
github.com/apache/arrow-go/v18 v18.0.1-0.20241212180703-82be143d7c30/go.mod h1:RNuWDIiGjq5nndL2PyQrndUy9nMLwheA3uWaAV7fe4U=
github.com/apache/arrow-go/v18 v18.2.0 h1:QhWqpgZMKfWOniGPhbUxrHohWnooGURqL2R2Gg4SO1Q=
github.com/apache/arrow-go/v18 v18.2.0/go.mod h1:Ic/01WSwGJWRrdAZcxjBZ5hbApNJ28K96jGYaxzzGUc=
github.com/apache/thrift v0.21.0 h1:tdPmh/ptjE1IJnhbhrcl2++TauVjy242rkV/UzJChnE=
github.com/apache/thrift v0.21.0/go.mod h1:W1H8aR/QRtYNvrPeFXBtobyRkd0/YVhTc6i07XIAgDw=
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 h1:DklsrG3dyBCFEj5IhUbnKptjxatkF07cF2ak3yi77so=
@@ -86,8 +87,8 @@ github.com/go-openapi/swag v0.23.0 h1:vsEVJDUo2hPJ2tu0/Xc+4noaxyEffXNIs3cOULZ+Gr
github.com/go-openapi/swag v0.23.0/go.mod h1:esZ8ITTYEsH1V2trKHjAN8Ai7xHb8RV+YSZ577vPjgQ=
github.com/go-test/deep v1.0.8 h1:TDsG77qcSprGbC6vTN8OuXp5g+J+b5Pcguhf7Zt61VM=
github.com/go-test/deep v1.0.8/go.mod h1:5C2ZWiW0ErCdrYzpqxLbTX7MG14M9iiw8DgHncVwcsE=
github.com/goccy/go-json v0.10.4 h1:JSwxQzIqKfmFX1swYPpUThQZp/Ka4wzJdK0LWVytLPM=
github.com/goccy/go-json v0.10.4/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v5 v5.2.2 h1:Rl4B7itRWVtYIHFrSNd7vhTiz9UpLdi6gZhZ3wEeDy8=
@@ -96,8 +97,8 @@ github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/golang/snappy v0.0.4 h1:yAGX7huGHXlcLOEtBnF4w7FQwA26wojNCwOYAEhLjQM=
github.com/golang/snappy v0.0.4/go.mod h1:/XxbfmMg8lxefKM7IXC3fBNl/7bRcc72aCRzEWrmP2Q=
github.com/google/flatbuffers v24.3.25+incompatible h1:CX395cjN9Kke9mmalRoL3d81AtFUxJM+yDthflgJGkI=
github.com/google/flatbuffers v24.3.25+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/flatbuffers v25.2.10+incompatible h1:F3vclr7C3HpB1k9mxCGRMXq6FdUalZ6H/pNX4FP1v0Q=
github.com/google/flatbuffers v25.2.10+incompatible/go.mod h1:1AeVuKshWv4vARoZatz6mlQ0JxURH0Kv5+zNeJKJCa8=
github.com/google/gnostic-models v0.6.8 h1:yo/ABAfM5IMRsS1VnXjTBvUb61tFIHozhlYvRgGre9I=
github.com/google/gnostic-models v0.6.8/go.mod h1:5n7qKqH0f5wFt+aWF8CW6pZLLNOfYuF5OpfBSENuI8U=
github.com/google/go-cmp v0.5.9/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
@@ -122,8 +123,8 @@ github.com/gorilla/mux v1.8.1 h1:TuBL49tXwgrFYWhqrNgrUNEY92u81SPhu7sTdzQEiWY=
github.com/gorilla/mux v1.8.1/go.mod h1:AKf9I4AEqPTmMytcMc0KkNouC66V3BtZ4qD5fmWSiMQ=
github.com/grafana/dskit v0.0.0-20241105154643-a6b453a88040 h1:IR+UNYHqaU31t8/TArJk8K/GlDwOyxMpGNkWCXeZ28g=
github.com/grafana/dskit v0.0.0-20241105154643-a6b453a88040/go.mod h1:SPLNCARd4xdjCkue0O6hvuoveuS1dGJjDnfxYe405YQ=
github.com/grafana/grafana-plugin-sdk-go v0.272.0 h1:TmPIG+6e3lYGzkyfUfCHuaMaaiwDbkCacTZ7V/JaSeg=
github.com/grafana/grafana-plugin-sdk-go v0.272.0/go.mod h1:i/9KH9y/6m5hkRnG3H6aR2nOMPbJUmvo4XNrHjI15cU=
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0 h1:qVdhLR+XkVdTQ2Sr7+VnRfGM8RMp8oPe25nghsSpQms=
github.com/grafana/grafana-plugin-sdk-go v0.274.1-0.20250318081012-21a7f15619b0/go.mod h1:jV+CTjXqXYuaz8FgSG7ALOib3sgiDo/00dfsQFVTSpM=
github.com/grafana/otel-profiling-go v0.5.1 h1:stVPKAFZSa7eGiqbYuG25VcqYksR6iWvF3YH66t4qL8=
github.com/grafana/otel-profiling-go v0.5.1/go.mod h1:ftN/t5A/4gQI19/8MoWurBEtC6gFw8Dns1sJZ9W4Tls=
github.com/grafana/pyroscope-go/godeltaprof v0.1.8 h1:iwOtYXeeVSAeYefJNaxDytgjKtUuKQbJqgAIjlnicKg=
@@ -163,10 +164,10 @@ github.com/kisielk/errcheck v1.5.0/go.mod h1:pFxgyoBC7bSaBwPgfKdkLd5X25qrDl4LWUI
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
github.com/klauspost/asmfmt v1.3.2 h1:4Ri7ox3EwapiOjCki+hw14RyKk201CN4rzyCJRFLpK4=
github.com/klauspost/asmfmt v1.3.2/go.mod h1:AG8TuvYojzulgDAMCnYn50l/5QV3Bs/tp6j0HLHbNSE=
github.com/klauspost/compress v1.17.11 h1:In6xLpyWOi1+C7tXUUWv2ot1QvBjxevKAaI6IXrJmUc=
github.com/klauspost/compress v1.17.11/go.mod h1:pMDklpSncoRMuLFrf1W9Ss9KT+0rH90U12bZKk7uwG0=
github.com/klauspost/cpuid/v2 v2.2.9 h1:66ze0taIn2H33fBvCkXuv9BmCwDfafmiIVpKV9kKGuY=
github.com/klauspost/cpuid/v2 v2.2.9/go.mod h1:rqkxqrZ1EhYM9G+hXH7YdowN5R5RGN6NK4QwQ3WMXF8=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/cpuid/v2 v2.2.10 h1:tBs3QSyvjDyFTq3uoc/9xFpCuOsJQFNPiAhYdw2skhE=
github.com/klauspost/cpuid/v2 v2.2.10/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE=
github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk=
github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
@@ -220,8 +221,8 @@ github.com/olekukonko/tablewriter v0.0.5 h1:P2Ga83D34wi1o9J6Wh1mRuqd4mF/x/lgBS7N
github.com/olekukonko/tablewriter v0.0.5/go.mod h1:hPp6KlRPjbx+hW8ykQs1w3UBbZlj6HuIJcUGPhkA7kY=
github.com/perimeterx/marshmallow v1.1.5 h1:a2LALqQ1BlHM8PZblsDdidgv1mWi1DgC2UmX50IvK2s=
github.com/perimeterx/marshmallow v1.1.5/go.mod h1:dsXbUu8CRzfYP5a87xpp0xq9S3u0Vchtcl8we9tYaXw=
github.com/pierrec/lz4/v4 v4.1.21 h1:yOVMLb6qSIDP67pl/5F7RepeKYu/VmTyEXvuMI5d9mQ=
github.com/pierrec/lz4/v4 v4.1.21/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pierrec/lz4/v4 v4.1.22 h1:cKFw6uJDK+/gfw5BcDL0JL5aBsAFdsIT18eRtLj7VIU=
github.com/pierrec/lz4/v4 v4.1.22/go.mod h1:gZWDp/Ze/IJXGXf23ltt2EXimqmTUXEy0GFuRQyBid4=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c h1:+mdjkGKdHQG3305AYmdv1U2eRNDiU2ErMBj1gwrq8eQ=
github.com/pkg/browser v0.0.0-20240102092130-5ac0b6a4141c/go.mod h1:7rwL4CYBLnjLxUqIJNnCWiEdr3bn6IUYi15bNlnbCCU=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
@@ -302,8 +303,8 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0 h1:sbiXRND
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0/go.mod h1:69uWxva0WgAA/4bu2Yy70SLDBwZXuQ6PbBpbsa5iZrQ=
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0 h1:D3htJISCUU/wOVlKwisVKancWm+2U4h9xDEaiMkiyRE=
go.opentelemetry.io/contrib/propagators/jaeger v1.34.0/go.mod h1:DAX1bsj+uDm2ZuOQH/RgZRx7RQZWyzV5W2WR/0UX8JA=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0 h1:Xx1N6cDr8iWy1Cz6OcY7oS0ACdt/6HDYTdu4KskuC7s=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.28.0/go.mod h1:iWS+NvC948FyfnJbVfPN9h/8+vr8CR2FPn6XsLRkvH8=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0 h1:VpYbyLrB5BS3blBCJMqHRIrbU4RlPnyFovR3La+1j4Q=
go.opentelemetry.io/contrib/samplers/jaegerremote v0.29.0/go.mod h1:XAJmM2MWhiIoTO4LCLBVeE8w009TmsYk6hq1UNdXs5A=
go.opentelemetry.io/otel v1.21.0/go.mod h1:QZzNPQPm1zLX4gZK4cMi+71eaorMSGT3A4znnUvNNEo=
go.opentelemetry.io/otel v1.35.0 h1:xKWKPxrxB6OtMCbmMY021CqC45J+3Onta9MqjhnusiQ=
go.opentelemetry.io/otel v1.35.0/go.mod h1:UEqy8Zp11hpkUrL73gSlELM0DupHoiq72dR+Zqel/+Y=
@@ -365,8 +366,8 @@ golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220811171246-fbc7d0a398ab/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.14.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.30.0 h1:QjkSwP/36a20jFYWkSue1YwXzLmsV5Gfq7Eiy72C1uc=
golang.org/x/sys v0.30.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
golang.org/x/sys v0.31.0 h1:ioabZlmFYtWhL+TRYpcnNlLwhyxaM9kWTDEmfnprqik=
golang.org/x/sys v0.31.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
@@ -390,12 +391,12 @@ gonum.org/v1/gonum v0.15.1 h1:FNy7N6OUZVUaWG9pTiD+jlhdQ3lMP+/LcTpJ6+a8sQ0=
gonum.org/v1/gonum v0.15.1/go.mod h1:eZTZuRFrzu5pcyjN5wJhcIhnUdNijYxX1T2IcrOGY0o=
google.golang.org/api v0.220.0 h1:3oMI4gdBgB72WFVwE1nerDD8W3HUOS4kypK6rRLbGns=
google.golang.org/api v0.220.0/go.mod h1:26ZAlY6aN/8WgpCzjPNy18QpYaz7Zgg1h0qe1GkZEmY=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489 h1:fCuMM4fowGzigT89NCIsW57Pk9k2D12MMi2ODn+Nk+o=
google.golang.org/genproto/googleapis/api v0.0.0-20250204164813-702378808489/go.mod h1:iYONQfRdizDB8JJBybql13nArx91jcUk7zCXEsOofM4=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6 h1:2duwAxN2+k0xLNpjnHTXoMUgnv6VPSp5fiqTuwSxjmI=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250207221924-e9438ea467c6/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.70.0 h1:pWFv03aZoHzlRKHWicjsZytKAiYCtNS0dHbXnIdq7jQ=
google.golang.org/grpc v1.70.0/go.mod h1:ofIJqVKDXx/JiXrwr2IG4/zwdH9txy3IlF40RmcJSQw=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a h1:nwKuGPlUAt+aR+pcrkfFRrTU1BVrSmYyYMxYbUIVHr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250218202821-56aae31c358a/go.mod h1:3kWAYMk1I75K4vykHtKt2ycnOgpA6974V7bREqbsenU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b h1:FQtJ1MxbXoIIrZHZ33M+w5+dAP9o86rgpjoKr/ZmT7k=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250212204824-5a70512c5d8b/go.mod h1:8BS3B93F/U1juMFq9+EDk+qOT5CO1R9IzXxG3PTqiRk=
google.golang.org/grpc v1.71.0 h1:kF77BGdPTQ4/JZWMlb9VpJ5pa25aqvVqogsxNHHdeBg=
google.golang.org/grpc v1.71.0/go.mod h1:H0GRtasmQOh9LkFoCPDu3ZrwUtD1YGE+b2vYBYd/8Ec=
google.golang.org/protobuf v1.36.5 h1:tPhr+woSbjfYvY6/GPufUoYizxw1cF/yFoxJ2fmpwlM=
google.golang.org/protobuf v1.36.5/go.mod h1:9fA7Ob0pmnwhb644+1+CVWFRbNajQ6iRojtC/QF5bRE=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
pkg/registry/apis/provisioning/apifmt/error.go
+109
View File
@@ -0,0 +1,109 @@
// apifmt aims to provide a Kubernetes-compatible way to format text.
package apifmt
import (
"errors"
"fmt"
"net/http"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
var (
_ error = (*fmtError)(nil)
_ apierrors.APIStatus = (*fmtError)(nil)
)
type fmtError struct {
inner error
str string
innerStatusErr apierrors.APIStatus
initInnerStatusErr bool
}
func (e *fmtError) Error() string {
return e.str
}
// Status returns the status that is closest in the tree, in a depth-first search.
func (e *fmtError) Status() metav1.Status {
if !e.initInnerStatusErr {
if status, ok := e.inner.(apierrors.APIStatus); ok || errors.As(e.inner, &status) {
e.innerStatusErr = status
}
e.initInnerStatusErr = true
}
status := metav1.Status{
Message: e.str,
Code: http.StatusInternalServerError,
Reason: metav1.StatusReasonInternalError,
Status: metav1.StatusFailure,
}
if e.innerStatusErr != nil {
s := e.innerStatusErr.Status()
status.Code, status.Reason, status.Status, status.Details = s.Code, s.Reason, s.Status, s.Details
}
return status
}
func (e *fmtError) Unwrap() error {
return e.inner
}
func (e *fmtError) Is(target error) bool {
if e.initInnerStatusErr && e.innerStatusErr != nil {
// If we already know the inner status, we can speed up the Is check for apierrors Is functions. These are the most common case.
if err, ok := e.innerStatusErr.(error); ok {
return errors.Is(err, target)
}
}
return errors.Is(e.inner, target)
}
// Errorf acts like `fmt.Errorf`. Use `%w` to wrap a specific error.
// The returned error will propagate the inner `metav1.Status`, if one exists. Otherwise, an HTTP 500 Internal Server Error will be returned.
// If multiple errors are passed, they will be joined with `errors.Join`, just like `fmt.Errorf`.
func Errorf(format string, args ...any) *fmtError {
// We go via Errorf to only give the %w errors as inner errors.
wrapped := fmt.Errorf(format, args...)
str := wrapped.Error()
err := unwrap(wrapped)
return &fmtError{
inner: err,
str: str,
}
}
// unwrap returns the inner error of an error, if it exists.
// If multiple errors are present, it will errors.Join them.
func unwrap(err error) error {
type singleUnwrapper interface {
Unwrap() error
}
type multiUnwrapper interface {
Unwrap() []error
}
if err == nil {
return nil
}
if e, ok := err.(singleUnwrapper); ok {
return e.Unwrap()
}
if e, ok := err.(multiUnwrapper); ok {
errs := e.Unwrap()
if len(errs) == 0 {
return err
}
if len(errs) == 1 && errs[0] != nil {
return errs[0]
}
return errors.Join(errs...)
}
return err
}
pkg/registry/apis/provisioning/apifmt/error_test.go
+97
View File
@@ -0,0 +1,97 @@
package apifmt_test
import (
"errors"
"fmt"
"net/http"
"testing"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/apifmt"
"github.com/stretchr/testify/assert"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
func TestErrorf(t *testing.T) {
t.Parallel()
for _, fmt := range []string{"1 %v 2 %v", "1 %w 2 %v", "1 %v 2 %w", "1 %w 2 %w"} {
t.Run("error string is formatted appropriately with fmt="+fmt, func(t *testing.T) {
t.Parallel()
err1 := errors.New("error1")
err2 := errors.New("error2")
err := apifmt.Errorf(fmt, err1, err2)
assert.Equal(t, "1 error1 2 error2", err.Error())
})
}
t.Run("no inner error defaults to internal server error", func(t *testing.T) {
t.Parallel()
err := apifmt.Errorf("nothing inside")
assert.True(t, apierrors.IsInternalError(err), "err is not internal error per apierrors")
assert.Equal(t, int32(http.StatusInternalServerError), err.Status().Code, ".Code")
assert.Equal(t, metav1.StatusReasonInternalError, err.Status().Reason, ".Reason")
assert.Equal(t, metav1.StatusFailure, err.Status().Status, ".Status")
})
t.Run("non-apistatus inner error defaults to internal server error", func(t *testing.T) {
t.Parallel()
inner := errors.New("an inner error")
err := apifmt.Errorf("%w", inner)
assert.True(t, apierrors.IsInternalError(err), "err is not internal error per apierrors")
assert.Equal(t, int32(http.StatusInternalServerError), err.Status().Code, ".Code")
assert.Equal(t, metav1.StatusReasonInternalError, err.Status().Reason, ".Reason")
assert.Equal(t, metav1.StatusFailure, err.Status().Status, ".Status")
})
t.Run("apistatus inner error is used for status", func(t *testing.T) {
t.Parallel()
inner := apierrors.NewBadRequest("bad request")
err := apifmt.Errorf("%w", inner)
assert.Equal(t, inner.Status(), err.Status(), "err.Status()")
})
t.Run("message is used with inner apistatus error", func(t *testing.T) {
t.Parallel()
inner := apierrors.NewBadRequest("bad request")
err := apifmt.Errorf("context here: %w", inner)
status := inner.Status()
status.Message = "context here: bad request"
assert.Equal(t, status, err.Status(), "err.Status()")
assert.Equal(t, "context here: bad request", err.Error(), "err.Error()")
})
t.Run("deep apierror is used", func(t *testing.T) {
t.Parallel()
inner := apierrors.NewBadRequest("bad request")
wrapped := fmt.Errorf("%w", inner)
wrapped = fmt.Errorf("%w", wrapped)
err := apifmt.Errorf("%w", wrapped)
assert.Equal(t, inner.Status(), err.Status(), "err.Status()")
})
t.Run("deep error in multi-unwrap wrapper's apierror is used", func(t *testing.T) {
t.Parallel()
inner := apierrors.NewBadRequest("bad request")
anotherError := errors.New("not an apierror")
wrapped := errors.Join(fmt.Errorf("this is cool: %w", anotherError), fmt.Errorf("another one: %w", errors.Join(anotherError, inner, anotherError)))
err := apifmt.Errorf("%w", wrapped)
status := inner.Status()
status.Message = "this is cool: not an apierror\nanother one: not an apierror\nbad request\nnot an apierror"
assert.Equal(t, status, err.Status(), "err.Status()")
})
}
pkg/registry/apis/provisioning/controller/finalizers.go
+142
View File
@@ -0,0 +1,142 @@
package controller
import (
"sort"
"strings"
"golang.org/x/net/context"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/dynamic"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/utils"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
)
// RemoveOrphanResourcesFinalizer removes everything this repo created
const RemoveOrphanResourcesFinalizer = "remove-orphan-resources"
// ReleaseOrphanResourcesFinalizer removes the metadata for anything this repo created
const ReleaseOrphanResourcesFinalizer = "release-orphan-resources"
// CleanFinalizer calls the "OnDelete" function for resource
const CleanFinalizer = "cleanup"
type finalizer struct {
lister resources.ResourceLister
clientFactory *resources.ClientFactory
}
func (f *finalizer) process(ctx context.Context,
repo repository.Repository,
finalizers []string,
) error {
logger := logging.FromContext(ctx)
for _, finalizer := range finalizers {
switch finalizer {
case CleanFinalizer:
// NOTE: the controller loop will never get run unless a finalizer is set
hooks, ok := repo.(repository.Hooks)
if ok {
if err := hooks.OnDelete(ctx); err != nil {
logger.Warn("Error running deletion hooks", "err", err)
}
}
case ReleaseOrphanResourcesFinalizer:
err := f.processExistingItems(ctx, repo.Config(),
func(client dynamic.ResourceInterface, item *provisioning.ResourceListItem) error {
_, err := client.Patch(ctx, item.Name, types.JSONPatchType, []byte(`[
{"op": "remove", "path": "/metadata/annotations/`+utils.AnnoKeyManagerKind+`" },
{"op": "remove", "path": "/metadata/annotations/`+utils.AnnoKeyManagerIdentity+`" },
{"op": "remove", "path": "/metadata/annotations/`+utils.AnnoKeySourcePath+`" },
{"op": "remove", "path": "/metadata/annotations/`+utils.AnnoKeySourceChecksum+`" }
]`), v1.PatchOptions{})
return err
})
if err != nil {
return err
}
case RemoveOrphanResourcesFinalizer:
err := f.processExistingItems(ctx, repo.Config(),
func(client dynamic.ResourceInterface, item *provisioning.ResourceListItem) error {
return client.Delete(ctx, item.Name, v1.DeleteOptions{})
})
if err != nil {
return err
}
default:
logger.Warn("skipping unknown finalizer", "finalizer", finalizer)
}
}
return nil
}
// internal iterator to walk the existing items
func (f *finalizer) processExistingItems(
ctx context.Context,
repo *provisioning.Repository,
cb func(client dynamic.ResourceInterface, item *provisioning.ResourceListItem) error,
) error {
logger := logging.FromContext(ctx)
clients, err := f.clientFactory.Clients(ctx, repo.Namespace)
if err != nil {
return err
}
items, err := f.lister.List(ctx, repo.Namespace, repo.Name)
if err != nil {
logger.Warn("error listing resources", "error", err)
return err
}
// Safe deletion order
sortResourceListForDeletion(items)
count := 0
errors := 0
for _, item := range items.Items {
res, _, err := clients.ForResource(schema.GroupVersionResource{
Group: item.Group,
Resource: item.Resource,
})
if err != nil {
return err
}
err = cb(res, &item)
if err != nil {
logger.Warn("error processing item", "name", item.Name, "error", err)
errors++
} else {
count++
}
}
logger.Info("processed orphan items", "items", count, "errors", errors)
return nil
}
func sortResourceListForDeletion(list *provisioning.ResourceList) {
// FIXME: this code should be simplified once unified storage folders support recursive deletion
// Sort by the following logic:
// - Put folders at the end so that we empty them first.
// - Sort folders by depth so that we remove the deepest first
sort.Slice(list.Items, func(i, j int) bool {
switch {
case list.Items[i].Group != folders.RESOURCE:
return true
case list.Items[j].Group != folders.RESOURCE:
return false
default:
return len(strings.Split(list.Items[i].Path, "/")) > len(strings.Split(list.Items[j].Path, "/"))
}
})
}
pkg/registry/apis/provisioning/controller/repository.go
+532
View File
@@ -0,0 +1,532 @@
package controller
import (
"context"
"encoding/json"
"errors"
"fmt"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
utilruntime "k8s.io/apimachinery/pkg/util/runtime"
"k8s.io/apimachinery/pkg/util/wait"
"k8s.io/apiserver/pkg/endpoints/request"
"k8s.io/client-go/tools/cache"
"k8s.io/client-go/util/workqueue"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/identity"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
client "github.com/grafana/grafana/pkg/generated/clientset/versioned/typed/provisioning/v0alpha1"
informer "github.com/grafana/grafana/pkg/generated/informers/externalversions/provisioning/v0alpha1"
listers "github.com/grafana/grafana/pkg/generated/listers/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/secrets"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
)
type RepoGetter interface {
// Given a repository configuration, return it as a repository instance
// This will only error for un-recoverable system errors
// the repository instance may or may not be valid/healthy
AsRepository(ctx context.Context, cfg *provisioning.Repository) (repository.Repository, error)
}
type RepositoryTester interface {
TestRepository(ctx context.Context, repo repository.Repository) (*provisioning.TestResults, error)
}
const loggerName = "provisioning-repository-controller"
const (
maxAttempts = 3
)
type queueItem struct {
key string
obj interface{}
attempts int
}
// RepositoryController controls how and when CRD is established.
type RepositoryController struct {
client client.ProvisioningV0alpha1Interface
resourceLister resources.ResourceLister
repoLister listers.RepositoryLister
repoSynced cache.InformerSynced
parsers *resources.ParserFactory
logger logging.Logger
secrets secrets.Service
dualwrite dualwrite.Service
jobs jobs.Queue
finalizer *finalizer
// Converts config to instance
repoGetter RepoGetter
tester RepositoryTester
// To allow injection for testing.
processFn func(item *queueItem) error
enqueueRepository func(obj any)
keyFunc func(obj any) (string, error)
queue workqueue.TypedRateLimitingInterface[*queueItem]
}
// NewRepositoryController creates new RepositoryController.
func NewRepositoryController(
provisioningClient client.ProvisioningV0alpha1Interface,
repoInformer informer.RepositoryInformer,
repoGetter RepoGetter,
resourceLister resources.ResourceLister,
parsers *resources.ParserFactory,
tester RepositoryTester,
jobs jobs.Queue,
secrets secrets.Service,
dualwrite dualwrite.Service,
) (*RepositoryController, error) {
rc := &RepositoryController{
client: provisioningClient,
resourceLister: resourceLister,
repoLister: repoInformer.Lister(),
repoSynced: repoInformer.Informer().HasSynced,
queue: workqueue.NewTypedRateLimitingQueueWithConfig(
workqueue.DefaultTypedControllerRateLimiter[*queueItem](),
workqueue.TypedRateLimitingQueueConfig[*queueItem]{
Name: "provisioningRepositoryController",
},
),
repoGetter: repoGetter,
parsers: parsers,
finalizer: &finalizer{
lister: resourceLister,
clientFactory: parsers.ClientFactory,
},
tester: tester,
jobs: jobs,
logger: logging.DefaultLogger.With("logger", loggerName),
secrets: secrets,
dualwrite: dualwrite,
}
_, err := repoInformer.Informer().AddEventHandler(cache.ResourceEventHandlerFuncs{
AddFunc: rc.enqueue,
UpdateFunc: func(oldObj, newObj interface{}) {
rc.enqueue(newObj)
},
})
if err != nil {
return nil, err
}
rc.processFn = rc.process
rc.enqueueRepository = rc.enqueue
rc.keyFunc = repoKeyFunc
return rc, nil
}
func repoKeyFunc(obj any) (string, error) {
repo, ok := obj.(*provisioning.Repository)
if !ok {
return "", fmt.Errorf("expected a Repository but got %T", obj)
}
return cache.DeletionHandlingMetaNamespaceKeyFunc(repo)
}
// Run starts the RepositoryController.
func (rc *RepositoryController) Run(ctx context.Context, workerCount int) {
defer utilruntime.HandleCrash()
defer rc.queue.ShutDown()
logger := rc.logger
ctx = logging.Context(ctx, logger)
logger.Info("Starting RepositoryController")
defer logger.Info("Shutting down RepositoryController")
if !cache.WaitForCacheSync(ctx.Done(), rc.repoSynced) {
return
}
logger.Info("Starting workers", "count", workerCount)
for i := 0; i < workerCount; i++ {
go wait.UntilWithContext(ctx, rc.runWorker, time.Second)
}
logger.Info("Started workers")
<-ctx.Done()
logger.Info("Shutting down workers")
}
func (rc *RepositoryController) runWorker(ctx context.Context) {
for rc.processNextWorkItem(ctx) {
}
}
func (rc *RepositoryController) enqueue(obj interface{}) {
key, err := rc.keyFunc(obj)
if err != nil {
utilruntime.HandleError(fmt.Errorf("couldn't get key for object: %v", err))
return
}
item := queueItem{key: key, obj: obj}
rc.queue.Add(&item)
}
// processNextWorkItem deals with one key off the queue.
// It returns false when it's time to quit.
func (rc *RepositoryController) processNextWorkItem(ctx context.Context) bool {
item, quit := rc.queue.Get()
if quit {
return false
}
defer rc.queue.Done(item)
// TODO: should we move tracking work to trace ids instead?
logger := logging.FromContext(ctx).With("work_key", item.key)
logger.Info("RepositoryController processing key")
err := rc.processFn(item)
if err == nil {
rc.queue.Forget(item)
return true
}
item.attempts++
logger = logger.With("error", err, "attempts", item.attempts)
logger.Error("RepositoryController failed to process key")
if item.attempts >= maxAttempts {
logger.Error("RepositoryController failed too many times")
rc.queue.Forget(item)
return true
}
if !apierrors.IsServiceUnavailable(err) {
logger.Info("RepositoryController will not retry")
rc.queue.Forget(item)
return true
} else {
logger.Info("RepositoryController will retry as service is unavailable")
}
utilruntime.HandleError(fmt.Errorf("%v failed with: %v", item, err))
rc.queue.AddRateLimited(item)
return true
}
func (rc *RepositoryController) handleDelete(ctx context.Context, obj *provisioning.Repository) error {
logger := logging.FromContext(ctx)
logger.Info("handle repository delete")
// Process any finalizers
if len(obj.Finalizers) > 0 {
repo, err := rc.repoGetter.AsRepository(ctx, obj)
if err != nil {
logger.Warn("unable to get repository for cleanup")
} else {
err := rc.finalizer.process(ctx, repo, obj.Finalizers)
if err != nil {
logger.Warn("error running finalizer", "err")
}
}
// remove the finalizers
_, err = rc.client.Repositories(obj.GetNamespace()).
Patch(ctx, obj.Name, types.JSONPatchType, []byte(`[
{ "op": "remove", "path": "/metadata/finalizers" }
]`), v1.PatchOptions{
FieldManager: "repository-controller",
})
return err // delete will be called again
}
return nil
}
func (rc *RepositoryController) shouldCheckHealth(obj *provisioning.Repository) bool {
if obj.Status.Health.Checked == 0 || obj.Generation != obj.Status.ObservedGeneration {
return true
}
healthAge := time.Since(time.UnixMilli(obj.Status.Health.Checked))
if obj.Status.Health.Healthy {
return healthAge > time.Minute*5 // when healthy, check every 5 mins
}
return healthAge > time.Minute // otherwise within a minute
}
func (rc *RepositoryController) runHealthCheck(ctx context.Context, repo repository.Repository) provisioning.HealthStatus {
logger := logging.FromContext(ctx)
logger.Info("running health check")
res, err := rc.tester.TestRepository(ctx, repo)
if err != nil {
res = &provisioning.TestResults{
Success: false,
Errors: []string{
"error running test repository",
err.Error(),
},
}
}
healthStatus := provisioning.HealthStatus{
Healthy: res.Success,
Checked: time.Now().UnixMilli(),
Message: res.Errors,
}
logger.Info("health check completed", "status", healthStatus)
return healthStatus
}
func (rc *RepositoryController) shouldResync(obj *provisioning.Repository) bool {
// don't trigger resync if a sync was never started
if obj.Status.Sync.Finished == 0 && obj.Status.Sync.State == "" {
return false
}
syncAge := time.Since(time.UnixMilli(obj.Status.Sync.Finished))
syncInterval := time.Duration(obj.Spec.Sync.IntervalSeconds) * time.Second
tolerance := time.Second
// HACK: how would this work in a multi-tenant world or under heavy load?
// It will start queueing up jobs and we will have to deal with that
pendingForTooLong := syncAge >= syncInterval/2 && obj.Status.Sync.State == provisioning.JobStatePending
isRunning := obj.Status.Sync.State == provisioning.JobStateWorking
return obj.Spec.Sync.Enabled && syncAge >= (syncInterval-tolerance) && !pendingForTooLong && !isRunning
}
func (rc *RepositoryController) runHooks(ctx context.Context, repo repository.Repository, obj *provisioning.Repository) (*provisioning.WebhookStatus, error) {
logger := logging.FromContext(ctx)
hooks, _ := repo.(repository.Hooks)
if hooks == nil || obj.Generation == obj.Status.ObservedGeneration {
return nil, nil
}
if obj.Status.ObservedGeneration < 1 {
logger.Info("handle repository create")
webhookStatus, err := hooks.OnCreate(ctx)
if err != nil {
return nil, fmt.Errorf("error running OnCreate: %w", err)
}
return webhookStatus, nil
}
logger.Info("handle repository spec update", "Generation", obj.Generation, "ObservedGeneration", obj.Status.ObservedGeneration)
webhookStatus, err := hooks.OnUpdate(ctx)
if err != nil {
return nil, fmt.Errorf("error running OnUpdate: %w", err)
}
return webhookStatus, nil
}
func (rc *RepositoryController) determineSyncStrategy(ctx context.Context, obj *provisioning.Repository, shouldResync bool, healthStatus provisioning.HealthStatus) *provisioning.SyncJobOptions {
logger := logging.FromContext(ctx)
switch {
case !obj.Spec.Sync.Enabled:
logger.Info("skip sync as it's disabled")
return nil
case !healthStatus.Healthy:
logger.Info("skip sync for unhealthy repository")
return nil
case dualwrite.IsReadingLegacyDashboardsAndFolders(ctx, rc.dualwrite):
logger.Info("skip sync as we are reading from legacy storage")
return nil
case healthStatus.Healthy != obj.Status.Health.Healthy:
logger.Info("repository became healthy, full resync")
return &provisioning.SyncJobOptions{}
case obj.Status.ObservedGeneration < 1:
logger.Info("full sync for new repository")
return &provisioning.SyncJobOptions{}
case obj.Generation != obj.Status.ObservedGeneration:
logger.Info("full sync for spec change")
return &provisioning.SyncJobOptions{}
case shouldResync:
logger.Info("incremental sync for sync interval")
return &provisioning.SyncJobOptions{Incremental: true}
default:
return nil
}
}
func (rc *RepositoryController) addSyncJob(ctx context.Context, obj *provisioning.Repository, syncOptions *provisioning.SyncJobOptions) error {
job, err := rc.jobs.Insert(ctx, &provisioning.Job{
ObjectMeta: v1.ObjectMeta{
Namespace: obj.Namespace,
},
Spec: provisioning.JobSpec{
Repository: obj.GetName(),
Action: provisioning.JobActionSync,
Pull: syncOptions,
},
})
if err != nil {
// FIXME: should we update the status of the repository if we fail to add the job?
return fmt.Errorf("error adding sync job: %w", err)
}
logging.FromContext(ctx).Info("sync job triggered", "job", job.Name)
return nil
}
func (rc *RepositoryController) patchStatus(ctx context.Context, obj *provisioning.Repository, patchOperations []map[string]interface{}) error {
if len(patchOperations) == 0 {
return nil
}
patch, err := json.Marshal(patchOperations)
if err != nil {
return fmt.Errorf("error encoding status patch: %w", err)
}
_, err = rc.client.Repositories(obj.GetNamespace()).
Patch(ctx, obj.Name, types.JSONPatchType, patch, v1.PatchOptions{}, "status")
if err != nil {
return fmt.Errorf("error applying status patch: %w", err)
}
return nil
}
func (rc *RepositoryController) determineSyncStatus(obj *provisioning.Repository, syncOptions *provisioning.SyncJobOptions) *provisioning.SyncStatus {
const unhealthyMessage = "Repository is unhealthy"
hasUnhealthyMessage := len(obj.Status.Sync.Message) > 0 && obj.Status.Sync.Message[0] == unhealthyMessage
switch {
case syncOptions != nil:
return &provisioning.SyncStatus{
State: provisioning.JobStatePending,
LastRef: obj.Status.Sync.LastRef,
Started: time.Now().UnixMilli(),
}
case obj.Status.Health.Healthy && hasUnhealthyMessage: // if the repository is healthy and the message is set, clear it
// FIXME: is this the clearest way to do this? Should we introduce another status or way of way of handling more
// specific errors?
return &provisioning.SyncStatus{
LastRef: obj.Status.Sync.LastRef,
}
case !obj.Status.Health.Healthy && !hasUnhealthyMessage: // if the repository is unhealthy and the message is not already set, set it
return &provisioning.SyncStatus{
State: provisioning.JobStateError,
Message: []string{unhealthyMessage},
LastRef: obj.Status.Sync.LastRef,
}
default:
return nil
}
}
//nolint:gocyclo
func (rc *RepositoryController) process(item *queueItem) error {
logger := rc.logger.With("key", item.key)
namespace, name, err := cache.SplitMetaNamespaceKey(item.key)
if err != nil {
return err
}
obj, err := rc.repoLister.Repositories(namespace).Get(name)
switch {
case apierrors.IsNotFound(err):
return errors.New("repository not found in cache")
case err != nil:
return err
}
ctx, _, err := identity.WithProvisioningIdentity(context.Background(), namespace)
if err != nil {
return err
}
ctx = request.WithNamespace(ctx, namespace)
logger = logger.WithContext(ctx)
if obj.DeletionTimestamp != nil {
return rc.handleDelete(ctx, obj)
}
shouldResync := rc.shouldResync(obj)
shouldCheckHealth := rc.shouldCheckHealth(obj)
hasSpecChanged := obj.Generation != obj.Status.ObservedGeneration
patchOperations := []map[string]interface{}{}
// Determine the main triggering condition
switch {
case hasSpecChanged:
logger.Info("spec changed", "Generation", obj.Generation, "ObservedGeneration", obj.Status.ObservedGeneration)
patchOperations = append(patchOperations, map[string]interface{}{
"op": "replace",
"path": "/status/observedGeneration",
"value": obj.Generation,
})
case shouldResync:
logger.Info("sync interval triggered", "sync_interval", time.Duration(obj.Spec.Sync.IntervalSeconds)*time.Second, "sync_status", obj.Status.Sync)
case shouldCheckHealth:
logger.Info("health is stale", "health_status", obj.Status.Health.Healthy)
default:
logger.Info("skipping as conditions are not met", "status", obj.Status, "generation", obj.Generation, "sync_spec", obj.Spec.Sync)
return nil
}
repo, err := rc.repoGetter.AsRepository(ctx, obj)
if err != nil {
return fmt.Errorf("unable to create repository from configuration: %w", err)
}
healthStatus := obj.Status.Health
if shouldCheckHealth {
healthStatus = rc.runHealthCheck(ctx, repo)
patchOperations = append(patchOperations, map[string]interface{}{
"op": "replace",
"path": "/status/health",
"value": healthStatus,
})
}
// Run hooks
webhookStatus, err := rc.runHooks(ctx, repo, obj)
switch {
case err != nil:
return err
case webhookStatus != nil:
patchOperations = append(patchOperations, map[string]interface{}{
"op": "replace",
"path": "/status/webhook",
"value": webhookStatus,
})
}
// determine the sync strategy and sync status to apply
syncOptions := rc.determineSyncStrategy(ctx, obj, shouldResync, healthStatus)
if syncStatus := rc.determineSyncStatus(obj, syncOptions); syncStatus != nil {
patchOperations = append(patchOperations, map[string]interface{}{
"op": "replace",
"path": "/status/sync",
"value": syncStatus,
})
}
// Apply all patch operations
if err := rc.patchStatus(ctx, obj, patchOperations); err != nil {
return err
}
// Trigger sync job after we have applied all patch operations
if syncOptions != nil {
if err := rc.addSyncJob(ctx, obj, syncOptions); err != nil {
return err
}
}
return nil
}
pkg/registry/apis/provisioning/export.go
+83
View File
@@ -0,0 +1,83 @@
package provisioning
import (
"context"
"net/http"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/registry/rest"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
type exportConnector struct {
repoGetter RepoGetter
jobs jobs.Queue
}
func (*exportConnector) New() runtime.Object {
return &provisioning.Job{}
}
func (*exportConnector) Destroy() {}
func (*exportConnector) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (c *exportConnector) ProducesObject(verb string) any {
return c.New()
}
func (*exportConnector) ConnectMethods() []string {
return []string{http.MethodPost}
}
func (*exportConnector) NewConnectOptions() (runtime.Object, bool, string) {
return nil, false, ""
}
func (c *exportConnector) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
repo, err := c.repoGetter.GetRepository(ctx, name)
if err != nil {
return nil, err
}
cfg := repo.Config()
if err := repository.IsWriteAllowed(cfg, ""); err != nil {
return nil, err
}
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
options := &provisioning.ExportJobOptions{}
if err := unmarshalJSON(r, defaultMaxBodySize, options); err != nil {
responder.Error(apierrors.NewBadRequest(err.Error()))
return
}
job, err := c.jobs.Insert(ctx, &provisioning.Job{
ObjectMeta: v1.ObjectMeta{
Namespace: cfg.Namespace,
},
Spec: provisioning.JobSpec{
Action: provisioning.JobActionExport,
Repository: cfg.Name,
Push: options,
},
})
if err != nil {
responder.Error(err)
} else {
responder.Object(http.StatusAccepted, job)
}
}), 30*time.Second), nil
}
var (
_ rest.Connecter = (*exportConnector)(nil)
_ rest.Storage = (*exportConnector)(nil)
_ rest.StorageMetadata = (*exportConnector)(nil)
)
pkg/registry/apis/provisioning/filepath.go
+16
View File
@@ -0,0 +1,16 @@
package provisioning
import (
"strings"
apierrors "k8s.io/apimachinery/pkg/api/errors"
)
func pathAfterPrefix(urlPath, prefix string) (string, error) {
idx := strings.Index(urlPath, prefix)
if idx == -1 {
return "", apierrors.NewBadRequest("invalid request path")
}
return strings.TrimPrefix(urlPath[idx+len(prefix):], "/"), nil
}
pkg/registry/apis/provisioning/filepath_test.go
+79
View File
@@ -0,0 +1,79 @@
package provisioning
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestPathAfterPrefix(t *testing.T) {
tests := []struct {
name string
urlPath string
prefix string
want string
expectError bool
}{
{
name: "basic path with prefix",
urlPath: "/a/b/prefix/c",
prefix: "/prefix",
want: "c",
expectError: false,
},
{
name: "path with multiple prefix occurrences",
urlPath: "/a/prefix/b/prefix/c",
prefix: "/prefix",
want: "b/prefix/c",
expectError: false,
},
{
name: "path without prefix",
urlPath: "/a/b/c",
prefix: "/prefix",
want: "",
expectError: true,
},
{
name: "empty path",
urlPath: "",
prefix: "/prefix",
want: "",
expectError: true,
},
{
name: "empty prefix",
urlPath: "/a/b/c/d",
prefix: "",
want: "a/b/c/d",
expectError: false,
},
{
name: "prefix at start of path",
urlPath: "/prefix/rest/of/path",
prefix: "/prefix",
want: "rest/of/path",
expectError: false,
},
{
name: "prefix in middle with special chars",
urlPath: "/a/b-c/prefix/d_e/f",
prefix: "/prefix",
want: "d_e/f",
expectError: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := pathAfterPrefix(tt.urlPath, tt.prefix)
if tt.expectError {
require.Error(t, err)
} else {
require.NoError(t, err)
require.Equal(t, tt.want, got)
}
})
}
}
pkg/registry/apis/provisioning/files.go
+386
View File
@@ -0,0 +1,386 @@
package provisioning
import (
"context"
"errors"
"fmt"
"net/http"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/registry/rest"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/apis/common/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
const (
// Files endpoint max size for dashboards etc (5MB)
filesMaxBodySize = 5 * 1024 * 1024
)
type filesConnector struct {
getter RepoGetter
parsers *resources.ParserFactory
}
func (*filesConnector) New() runtime.Object {
// This is added as the "ResponseType" regardless what ProducesObject() returns
return &provisioning.ResourceWrapper{}
}
func (*filesConnector) Destroy() {}
func (*filesConnector) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (*filesConnector) ProducesObject(verb string) any {
return &provisioning.ResourceWrapper{}
}
func (*filesConnector) ConnectMethods() []string {
return []string{http.MethodGet, http.MethodPut, http.MethodPost, http.MethodDelete}
}
func (*filesConnector) NewConnectOptions() (runtime.Object, bool, string) {
return nil, true, "" // true adds the {path} component
}
// TODO: document the synchronous write and delete on the API Spec
// TODO: Move dual write logic to `resources` package and keep this connector simple
func (s *filesConnector) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
logger := logging.FromContext(ctx).With("logger", "files-connector", "repository_name", name)
ctx = logging.Context(ctx, logger)
repo, err := s.getter.GetRepository(ctx, name)
if err != nil {
logger.Debug("failed to find repository", "error", err)
return nil, err
}
reader, ok := repo.(repository.Reader)
if !ok {
return nil, apierrors.NewBadRequest("repository does not support read")
}
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
query := r.URL.Query()
ref := query.Get("ref")
message := query.Get("message")
logger := logger.With("url", r.URL.Path, "ref", ref, "message", message)
ctx := logging.Context(r.Context(), logger)
filePath, err := pathAfterPrefix(r.URL.Path, fmt.Sprintf("/%s/files", name))
if err != nil {
responder.Error(apierrors.NewBadRequest(err.Error()))
return
}
if err := resources.IsPathSupported(filePath); err != nil {
responder.Error(apierrors.NewBadRequest(err.Error()))
return
}
isDir := safepath.IsDir(filePath)
if r.Method == http.MethodGet && isDir {
// TODO: Implement folder navigation
if len(filePath) > 0 {
responder.Error(apierrors.NewBadRequest("folder navigation not yet supported"))
return
}
// TODO: Add pagination
rsp, err := reader.ReadTree(ctx, ref)
if err != nil {
responder.Error(err)
return
}
files := &provisioning.FileList{}
for _, v := range rsp {
if !v.Blob {
continue // folder item
}
files.Items = append(files.Items, provisioning.FileItem{
Path: v.Path,
Size: v.Size,
Hash: v.Hash,
})
}
responder.Object(http.StatusOK, files)
return
}
if filePath == "" {
responder.Error(apierrors.NewBadRequest("path is required"))
return
}
// TODO: Implement folder delete
if r.Method == http.MethodDelete && isDir {
responder.Error(apierrors.NewBadRequest("folder navigation not yet supported"))
return
}
var obj *provisioning.ResourceWrapper
code := http.StatusOK
switch r.Method {
case http.MethodGet:
code, obj, err = s.doRead(ctx, reader, filePath, ref)
case http.MethodPost:
obj, err = s.doWrite(ctx, false, repo, filePath, ref, message, r)
case http.MethodPut:
// TODO: document in API specification
if isDir {
err = apierrors.NewMethodNotSupported(provisioning.RepositoryResourceInfo.GroupResource(), r.Method)
} else {
obj, err = s.doWrite(ctx, true, repo, filePath, ref, message, r)
}
case http.MethodDelete:
// TODO: limit file size
obj, err = s.doDelete(ctx, repo, filePath, ref, message)
default:
err = apierrors.NewMethodNotSupported(provisioning.RepositoryResourceInfo.GroupResource(), r.Method)
}
if err != nil {
logger.Debug("got an error after processing request", "error", err)
responder.Error(err)
return
}
// something failed
if len(obj.Errors) > 0 {
code = http.StatusInternalServerError
}
logger.Debug("request resulted in valid object", "object", obj)
responder.Object(code, obj)
}), 30*time.Second), nil
}
func (s *filesConnector) doRead(ctx context.Context, repo repository.Reader, path string, ref string) (int, *provisioning.ResourceWrapper, error) {
info, err := repo.Read(ctx, path, ref)
if err != nil {
return 0, nil, err
}
parser, err := s.parsers.GetParser(ctx, repo)
if err != nil {
return 0, nil, err
}
parsed, err := parser.Parse(ctx, info, true)
if err != nil {
return 0, nil, err
}
// GVR will exist for anything we can actually save
// TODO: Add known error in parser for unsupported resource
if parsed.GVR == nil {
if parsed.GVK != nil {
//nolint:govet
parsed.Errors = append(parsed.Errors, fmt.Errorf("unknown resource for Kind: %s", parsed.GVK.Kind))
} else {
parsed.Errors = append(parsed.Errors, fmt.Errorf("unknown resource"))
}
}
code := http.StatusOK
if len(parsed.Errors) > 0 {
code = http.StatusNotAcceptable
}
return code, parsed.AsResourceWrapper(), nil
}
func (s *filesConnector) doWrite(ctx context.Context, update bool, repo repository.Repository, path string, ref string, message string, req *http.Request) (*provisioning.ResourceWrapper, error) {
if err := repository.IsWriteAllowed(repo.Config(), ref); err != nil {
return nil, err
}
writer, ok := repo.(repository.ReaderWriter)
if !ok {
return nil, apierrors.NewBadRequest("repository does not support read-writing")
}
parser, err := s.parsers.GetParser(ctx, writer)
if err != nil {
return nil, err
}
defer func() { _ = req.Body.Close() }()
if safepath.IsDir(path) {
return s.doCreateFolder(ctx, writer, path, ref, message, parser)
}
data, err := readBody(req, filesMaxBodySize)
if err != nil {
return nil, err
}
info := &repository.FileInfo{
Data: data,
Path: path,
Ref: ref,
}
// TODO: improve parser to parse out of reader
parsed, err := parser.Parse(ctx, info, true)
if err != nil {
if errors.Is(err, resources.ErrUnableToReadResourceBytes) {
return nil, apierrors.NewBadRequest("unable to read the request as a resource")
}
return nil, err
}
// GVR will exist for anything we can actually save
// TODO: Add known error in parser for unsupported resource
if parsed.GVR == nil {
return nil, apierrors.NewBadRequest("The payload does not map to a known resource")
}
// Do not write if any errors exist
if len(parsed.Errors) > 0 {
return parsed.AsResourceWrapper(), err
}
data, err = parsed.ToSaveBytes()
if err != nil {
return nil, err
}
if update {
err = writer.Update(ctx, path, ref, data, message)
} else {
err = writer.Create(ctx, path, ref, data, message)
}
if err != nil {
return nil, err
}
// Directly update the grafana database
// Behaves the same running sync after writing
if ref == "" {
if parsed.Existing == nil {
parsed.Upsert, err = parsed.Client.Create(ctx, parsed.Obj, metav1.CreateOptions{})
if err != nil {
parsed.Errors = append(parsed.Errors, err)
}
} else {
parsed.Upsert, err = parsed.Client.Update(ctx, parsed.Obj, metav1.UpdateOptions{})
if err != nil {
parsed.Errors = append(parsed.Errors, err)
}
}
}
return parsed.AsResourceWrapper(), err
}
func (s *filesConnector) doCreateFolder(ctx context.Context, repo repository.Writer, path string, ref string, message string, parser *resources.Parser) (*provisioning.ResourceWrapper, error) {
client, err := parser.Clients().Folder()
if err != nil {
return nil, err
}
manager := resources.NewFolderManager(repo, client)
// Now actually create the folder
if err := repo.Create(ctx, path, ref, nil, message); err != nil {
return nil, fmt.Errorf("failed to create folder: %w", err)
}
cfg := repo.Config()
wrap := &provisioning.ResourceWrapper{
Path: path,
Ref: ref,
Repository: provisioning.ResourceRepositoryInfo{
Type: cfg.Spec.Type,
Namespace: cfg.Namespace,
Name: cfg.Name,
Title: cfg.Spec.Title,
},
Resource: provisioning.ResourceObjects{
Action: provisioning.ResourceActionCreate,
},
}
if ref == "" {
folderName, err := manager.EnsureFolderPathExist(ctx, path)
if err != nil {
return nil, err
}
current, err := manager.GetFolder(ctx, folderName)
if err != nil && !apierrors.IsNotFound(err) {
return nil, err // unable to check if the folder exists
}
wrap.Resource.Upsert = v0alpha1.Unstructured{
Object: current.Object,
}
}
return wrap, nil
}
// Deletes a file from the repository and the Grafana database.
// If the path is a folder, it will return an error.
// If the file is not parsable, it will return an error.
func (s *filesConnector) doDelete(ctx context.Context, repo repository.Repository, path string, ref string, message string) (*provisioning.ResourceWrapper, error) {
if err := repository.IsWriteAllowed(repo.Config(), ref); err != nil {
return nil, err
}
// Read the existing value
access, ok := repo.(repository.ReaderWriter)
if !ok {
return nil, fmt.Errorf("repository is not read+writeable")
}
file, err := access.Read(ctx, path, ref)
if err != nil {
return nil, err // unable to read value
}
parser, err := s.parsers.GetParser(ctx, access)
if err != nil {
return nil, err // unable to read value
}
// TODO: document in API specification
// We can only delete parsable things
parsed, err := parser.Parse(ctx, file, false)
if err != nil {
return nil, err // unable to read value
}
parsed.Action = provisioning.ResourceActionDelete
wrap := parsed.AsResourceWrapper()
// Now delete the file
err = access.Delete(ctx, path, ref, message)
if err != nil {
return nil, err
}
// Delete the file in the grafana database
if ref == "" {
err = parsed.Client.Delete(ctx, parsed.Obj.GetName(), metav1.DeleteOptions{})
if apierrors.IsNotFound(err) {
err = nil // ignorable
}
}
return wrap, err
}
var (
_ rest.Storage = (*filesConnector)(nil)
_ rest.Connecter = (*filesConnector)(nil)
_ rest.StorageMetadata = (*filesConnector)(nil)
)
pkg/registry/apis/provisioning/history.go
+97
View File
@@ -0,0 +1,97 @@
package provisioning
import (
"context"
"fmt"
"net/http"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/registry/rest"
"github.com/grafana/grafana-app-sdk/logging"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
)
type historySubresource struct {
repoGetter RepoGetter
}
func (h *historySubresource) New() runtime.Object {
// This is added as the "ResponseType" regardless what ProducesObject() returns
return &provisioning.HistoryList{}
}
func (h *historySubresource) Destroy() {}
func (h *historySubresource) NamespaceScoped() bool {
return true
}
func (h *historySubresource) GetSingularName() string {
return "History"
}
func (h *historySubresource) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (h *historySubresource) ProducesObject(verb string) runtime.Object {
return &provisioning.HistoryList{}
}
func (h *historySubresource) ConnectMethods() []string {
return []string{http.MethodGet}
}
func (h *historySubresource) NewConnectOptions() (runtime.Object, bool, string) {
return nil, true, "" // true adds the {path} component
}
func (h *historySubresource) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
logger := logging.FromContext(ctx).With("logger", "history-subresource")
ctx = logging.Context(ctx, logger)
repo, err := h.repoGetter.GetRepository(ctx, name)
if err != nil {
logger.Debug("failed to find repository", "error", err)
return nil, err
}
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
versioned, ok := repo.(repository.Versioned)
if !ok {
responder.Error(apierrors.NewBadRequest("this repository does not support history"))
return
}
query := r.URL.Query()
ref := query.Get("ref")
filePath, err := pathAfterPrefix(r.URL.Path, fmt.Sprintf("/%s/history/", name))
if err != nil {
responder.Error(apierrors.NewBadRequest(err.Error()))
return
}
if err := resources.IsPathSupported(filePath); err != nil {
responder.Error(apierrors.NewBadRequest(err.Error()))
return
}
logger = logger.With("ref", ref, "path", filePath)
ctx = logging.Context(r.Context(), logger)
// TODO: Add history pagination
commits, err := versioned.History(ctx, filePath, ref)
if err != nil {
logger.Debug("failed to get history", "error", err)
responder.Error(err)
return
}
responder.Object(http.StatusOK, &provisioning.HistoryList{Items: commits})
}), 30*time.Second), nil
}
pkg/registry/apis/provisioning/jobs/driver.go
+211
View File
@@ -0,0 +1,211 @@
package jobs
import (
"context"
"errors"
"time"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/identity"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/apifmt"
"k8s.io/apiserver/pkg/endpoints/request"
)
// Store is an abstraction for the storage API.
// This exists to allow for unit testing.
type Store interface {
// Claim takes a job from storage, marks it as ours, and returns it.
//
// Any job which has not been claimed by another worker is fair game.
//
// If err is not nil, the job and rollback values are always nil.
// The err may be ErrNoJobs if there are no jobs to claim.
Claim(ctx context.Context) (job *provisioning.Job, rollback func(), err error)
// Complete marks a job as completed and moves it to the historic job store.
// When in the historic store, there is no more claim on the job.
Complete(ctx context.Context, job *provisioning.Job) error
// Cleanup should be called periodically to clean up abandoned jobs.
// An abandoned job is one that has been claimed by a worker, but the worker has not updated the job in a while.
Cleanup(ctx context.Context) error
// InsertNotifications returns a channel that will have a value sent to it when a new job is inserted.
// This is used to wake up the job driver when a new job is inserted.
InsertNotifications() chan struct{}
// Update saves the job back to the store.
Update(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error)
}
var _ Store = (*persistentStore)(nil)
// jobDriver drives jobs to completion and manages the job queue.
// There may be multiple jobDrivers running in parallel.
// The jobDriver deals with cleaning up upon death and ensuring that jobs remain claimable.
type jobDriver struct {
// Timeout for processing a job. This should be the same or less than a claim expiry.
timeout time.Duration
// CleanupInterval is the time between cleanup runs.
cleanupInterval time.Duration
// JobInterval is the time between job ticks. This should be relatively low.
jobInterval time.Duration
// Store is the job storage backend.
store Store
// RepoGetter lets us access repositories to pass to the worker.
repoGetter RepoGetter
// Workers process the job.
// Only the first worker who supports the job will process it; the rest are ignored.
workers []Worker
}
func NewJobDriver(
timeout, cleanupInterval, jobInterval time.Duration,
store Store,
repoGetter RepoGetter,
workers ...Worker,
) *jobDriver {
return &jobDriver{
timeout: timeout,
cleanupInterval: cleanupInterval,
jobInterval: jobInterval,
store: store,
repoGetter: repoGetter,
workers: workers,
}
}
// Run drives jobs to completion. This is a blocking function.
// It will run until the context is canceled.
// This is a thread-safe function; it may be called from multiple goroutines.
func (d *jobDriver) Run(ctx context.Context) {
cleanupTicker := time.NewTicker(d.cleanupInterval)
defer cleanupTicker.Stop()
jobTicker := time.NewTicker(d.jobInterval)
defer jobTicker.Stop()
logger := logging.FromContext(ctx).With("logger", "job-driver")
ctx = logging.Context(ctx, logger)
ctx, _, err := identity.WithProvisioningIdentity(ctx, "*") // "*" grants us access to all namespaces.
if err != nil {
logger.Error("failed to grant provisioning identity; this will panic!", "error", err)
panic("unreachable?: failed to grant provisioning identity: " + err.Error())
}
// Drive without waiting on startup.
d.startDriving(ctx)
for {
select {
case <-cleanupTicker.C:
if err := d.store.Cleanup(ctx); err != nil {
logger.Error("failed to cleanup jobs", "error", err)
}
case <-jobTicker.C:
d.startDriving(ctx)
case <-d.store.InsertNotifications():
d.startDriving(ctx)
}
}
}
func (d *jobDriver) startDriving(ctx context.Context) {
timeoutCtx, cancel := context.WithTimeout(ctx, d.timeout)
defer cancel()
for timeoutCtx.Err() == nil {
if err := d.drive(timeoutCtx); err != nil {
if !errors.Is(err, context.Canceled) && !errors.Is(err, ErrNoJobs) {
logging.FromContext(ctx).Error("failed to drive jobs", "error", err)
}
break
}
}
}
func (d *jobDriver) drive(ctx context.Context) error {
logger := logging.FromContext(ctx)
// Claim a job to work on.
job, rollback, err := d.store.Claim(ctx)
if err != nil {
return apifmt.Errorf("failed to claim job: %w", err)
}
// Ensure that the job is cleaned up if we fail to complete it.
// The rollback function does not care about cancellations.
defer rollback()
logger = logger.With("job", job.GetName(), "namespace", job.GetNamespace())
ctx = logging.Context(ctx, logger)
logger.Debug("claimed a job")
// Now that we have a job, we need to augment our namespace to grant ourselves permission to work on it.
// Incidentally, this also limits our permissions to only the namespace of the job.
ctx = request.WithNamespace(ctx, job.GetNamespace())
ctx, _, err = identity.WithProvisioningIdentity(ctx, job.GetNamespace())
if err != nil {
return apifmt.Errorf("failed to grant provisioning identity: %w", err)
}
// Process the job.
start := time.Now()
err = d.processJob(ctx, job) // NOTE: We pass in a pointer here such that the job status can be kept in Complete without re-fetching.
end := time.Now()
logger.Debug("job processed", "duration", end.Sub(start), "error", err)
if err != nil {
return apifmt.Errorf("failed to process job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
// Mark the job as completed.
if err := d.store.Complete(ctx, job); err != nil {
return apifmt.Errorf("failed to complete job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
logger.Debug("job completed")
return nil
}
func (d *jobDriver) processJob(ctx context.Context, job *provisioning.Job) error {
for _, worker := range d.workers {
if !worker.IsSupported(ctx, *job) {
continue
}
repo, err := d.repoGetter.GetRepository(ctx, job.Spec.Repository)
if err != nil {
return apifmt.Errorf("failed to get repository '%s': %w", job.Spec.Repository, err)
}
recorder := newJobProgressRecorder(d.onProgress(job))
err = worker.Process(ctx, repo, *job, recorder)
if err != nil {
return apifmt.Errorf("worker failed to process job: %w", err)
}
job.Status = recorder.Complete(ctx, err)
return nil
}
return apifmt.Errorf("no workers were registered to handle the job")
}
func (d *jobDriver) onProgress(job *provisioning.Job) ProgressFn {
return func(ctx context.Context, status provisioning.JobStatus) error {
logging.FromContext(ctx).Debug("job progress", "status", status)
job.Status = status
updated, err := d.store.Update(ctx, job)
if err != nil {
return apifmt.Errorf("failed to update job: %w", err)
}
*job = *updated
return nil
}
}
pkg/registry/apis/provisioning/jobs/export/folders.go
+96
View File
@@ -0,0 +1,96 @@
package export
import (
"context"
"errors"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
// FIXME: revise logging in this method
func (r *exportJob) loadFolders(ctx context.Context) error {
logger := r.logger
r.progress.SetMessage(ctx, "reading folder tree")
repoName := r.target.Config().Name
// TODO: should this be logging or message or both?
r.progress.SetMessage(ctx, "read folder tree from unified storage")
client, err := r.client.Folder()
if err != nil {
return err
}
rawList, err := client.List(ctx, metav1.ListOptions{Limit: 10000})
if err != nil {
return fmt.Errorf("failed to list folders: %w", err)
}
if rawList.GetContinue() != "" {
return fmt.Errorf("unable to list all folders in one request: %s", rawList.GetContinue())
}
for _, item := range rawList.Items {
err = r.folderTree.AddUnstructured(&item, repoName)
if err != nil {
r.progress.Record(ctx, jobs.JobResourceResult{
Name: item.GetName(),
Resource: folders.RESOURCE,
Group: folders.GROUP,
Error: err,
})
}
}
// create folders first is required so that empty folders exist when finished
r.progress.SetMessage(ctx, "write folders")
err = r.folderTree.Walk(ctx, func(ctx context.Context, folder resources.Folder) error {
p := folder.Path
if r.path != "" {
p = safepath.Join(r.path, p)
}
logger := logger.With("path", p)
result := jobs.JobResourceResult{
Name: folder.ID,
Resource: folders.RESOURCE,
Group: folders.GROUP,
Path: p,
}
_, err := r.target.Read(ctx, p, r.ref)
if err != nil && !(errors.Is(err, repository.ErrFileNotFound) || apierrors.IsNotFound(err)) {
result.Error = fmt.Errorf("failed to check if folder exists before writing: %w", err)
return result.Error
} else if err == nil {
logger.Info("folder already exists")
result.Action = repository.FileActionIgnored
r.progress.Record(ctx, result)
return nil
}
result.Action = repository.FileActionCreated
msg := fmt.Sprintf("export folder %s", p)
// Create with an empty body will make a folder (or .keep file if unsupported)
if err := r.target.Create(ctx, p, r.ref, nil, msg); err != nil {
result.Error = fmt.Errorf("failed to write folder in repo: %w", err)
r.progress.Record(ctx, result)
return result.Error
}
r.progress.Record(ctx, result)
return nil
})
if err != nil {
return fmt.Errorf("failed to write folders: %w", err)
}
return nil
}
pkg/registry/apis/provisioning/jobs/export/job.go
+45
View File
@@ -0,0 +1,45 @@
package export
import (
"context"
"github.com/grafana/grafana-app-sdk/logging"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
)
// ExportJob holds all context for a running job
type exportJob struct {
logger logging.Logger
client *resources.ResourceClients // Read from
target repository.ReaderWriter // Write to
namespace string
progress jobs.JobProgressRecorder
folderTree *resources.FolderTree
path string // from options (now clean+safe)
ref string // from options (only git)
keepIdentifier bool
}
func newExportJob(ctx context.Context,
target repository.ReaderWriter,
options provisioning.ExportJobOptions,
clients *resources.ResourceClients,
progress jobs.JobProgressRecorder,
) *exportJob {
return &exportJob{
namespace: target.Config().Namespace,
target: target,
client: clients,
logger: logging.FromContext(ctx),
progress: progress,
path: options.Path,
ref: options.Branch,
keepIdentifier: options.Identifier,
folderTree: resources.NewEmptyFolderTree(),
}
}
pkg/registry/apis/provisioning/jobs/export/resources.go
+149
View File
@@ -0,0 +1,149 @@
package export
import (
"context"
"encoding/json"
"fmt"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
"github.com/grafana/grafana/pkg/apimachinery/utils"
"github.com/grafana/grafana/pkg/infra/slugify"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
func (r *exportJob) loadResources(ctx context.Context) error {
kinds := []schema.GroupVersionResource{{
Group: dashboard.GROUP,
Resource: dashboard.DASHBOARD_RESOURCE,
Version: "v1alpha1",
}}
for _, kind := range kinds {
r.progress.SetMessage(ctx, fmt.Sprintf("reading %s resource", kind.Resource))
if err := r.loadResourcesFromAPIServer(ctx, kind); err != nil {
return fmt.Errorf("error loading %s %w", kind.Resource, err)
}
}
return nil
}
func (r *exportJob) loadResourcesFromAPIServer(ctx context.Context, kind schema.GroupVersionResource) error {
client, _, err := r.client.ForResource(kind)
if err != nil {
return err
}
var continueToken string
for {
list, err := client.List(ctx, metav1.ListOptions{Limit: 100, Continue: continueToken})
if err != nil {
return fmt.Errorf("error executing list: %w", err)
}
for _, item := range list.Items {
r.progress.Record(ctx, r.write(ctx, &item))
if err := r.progress.TooManyErrors(); err != nil {
return err
}
}
continueToken = list.GetContinue()
if continueToken == "" {
break
}
}
return nil
}
func (r *exportJob) write(ctx context.Context, obj *unstructured.Unstructured) jobs.JobResourceResult {
gvk := obj.GroupVersionKind()
result := jobs.JobResourceResult{
Name: obj.GetName(),
Resource: gvk.Kind,
Group: gvk.Group,
Action: repository.FileActionCreated,
}
if err := ctx.Err(); err != nil {
result.Error = fmt.Errorf("context error: %w", err)
return result
}
meta, err := utils.MetaAccessor(obj)
if err != nil {
result.Error = fmt.Errorf("extract meta accessor: %w", err)
return result
}
// Message from annotations
commitMessage := meta.GetMessage()
if commitMessage == "" {
g := meta.GetGeneration()
if g > 0 {
commitMessage = fmt.Sprintf("Generation: %d", g)
} else {
commitMessage = "exported from grafana"
}
}
name := meta.GetName()
manager, _ := meta.GetManagerProperties()
if manager.Identity == r.target.Config().GetName() {
result.Action = repository.FileActionIgnored
return result
}
title := meta.FindTitle("")
if title == "" {
title = name
}
folder := meta.GetFolder()
// Get the absolute path of the folder
fid, ok := r.folderTree.DirPath(folder, "")
if !ok {
// FIXME: Shouldn't this fail instead?
fid = resources.Folder{
Path: "__folder_not_found/" + slugify.Slugify(folder),
}
r.logger.Error("folder of item was not in tree of repository")
}
result.Path = fid.Path
// Clear the metadata
delete(obj.Object, "metadata")
if r.keepIdentifier {
meta.SetName(name) // keep the identifier in the metadata
}
body, err := json.MarshalIndent(obj.Object, "", " ")
if err != nil {
result.Error = fmt.Errorf("failed to marshal dashboard: %w", err)
return result
}
fileName := slugify.Slugify(title) + ".json"
if fid.Path != "" {
fileName = safepath.Join(fid.Path, fileName)
}
if r.path != "" {
fileName = safepath.Join(r.path, fileName)
}
err = r.target.Write(ctx, fileName, r.ref, body, commitMessage)
if err != nil {
result.Error = fmt.Errorf("failed to write file: %w", err)
}
return result
}
pkg/registry/apis/provisioning/jobs/export/worker.go
+124
View File
@@ -0,0 +1,124 @@
package export
import (
"context"
"errors"
"fmt"
"os"
"time"
"github.com/grafana/grafana-app-sdk/logging"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
gogit "github.com/grafana/grafana/pkg/registry/apis/provisioning/repository/go-git"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/secrets"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
)
type ExportWorker struct {
// Tempdir for repo clones
clonedir string
// required to create clients
clientFactory *resources.ClientFactory
// Check where values are currently saved
storageStatus dualwrite.Service
// Decrypt secrets in config
secrets secrets.Service
}
func NewExportWorker(clientFactory *resources.ClientFactory,
storageStatus dualwrite.Service,
secrets secrets.Service,
clonedir string,
) *ExportWorker {
return &ExportWorker{
clonedir,
clientFactory,
storageStatus,
secrets,
}
}
func (r *ExportWorker) IsSupported(ctx context.Context, job provisioning.Job) bool {
return job.Spec.Action == provisioning.JobActionExport
}
// Process will start a job
func (r *ExportWorker) Process(ctx context.Context, repo repository.Repository, job provisioning.Job, progress jobs.JobProgressRecorder) error {
options := job.Spec.Push
if options == nil {
return errors.New("missing export settings")
}
// Can write to external branch
err := repository.IsWriteAllowed(repo.Config(), options.Branch)
if err != nil {
return err
}
// Use the existing clone if already checked out
buffered, ok := repo.(*gogit.GoGitRepo)
if !ok && repo.Config().Spec.GitHub != nil {
progress.SetMessage(ctx, "clone target")
buffered, err = gogit.Clone(ctx, repo.Config(), gogit.GoGitCloneOptions{
Root: r.clonedir,
SingleCommitBeforePush: true,
// TODO: make this configurable
Timeout: 10 * time.Minute,
}, r.secrets, os.Stdout)
if err != nil {
return fmt.Errorf("unable to clone target: %w", err)
}
repo = buffered // send all writes to the buffered repo
defer func() {
if err := buffered.Remove(ctx); err != nil {
logging.FromContext(ctx).Error("failed to remove cloned repository after export", "err", err)
}
}()
options.Branch = "" // :( the branch is now baked into the repo
}
rw, ok := repo.(repository.ReaderWriter)
if !ok {
return errors.New("export job submitted targeting repository that is not a ReaderWriter")
}
clients, err := r.clientFactory.Clients(ctx, repo.Config().Namespace)
if err != nil {
return err
}
worker := newExportJob(ctx, rw, *options, clients, progress)
// Load and write all folders
progress.SetMessage(ctx, "start folder export")
err = worker.loadFolders(ctx)
if err != nil {
return err
}
progress.SetMessage(ctx, "start resource export")
err = worker.loadResources(ctx)
if err != nil {
return err
}
if buffered != nil {
progress.SetMessage(ctx, "push changes")
if err := buffered.Push(ctx, gogit.GoGitPushOptions{
// TODO: make this configurable
Timeout: 10 * time.Minute,
}, os.Stdout); err != nil {
return fmt.Errorf("error pushing changes: %w", err)
}
}
return nil
}
pkg/registry/apis/provisioning/jobs/migrate/folders.go
+113
View File
@@ -0,0 +1,113 @@
package migrate
import (
"context"
"errors"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/dashboard/legacy"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/storage/unified/parquet"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
var _ resource.BulkResourceWriter = (*folderReader)(nil)
type folderReader struct {
tree *resources.FolderTree
targetRepoName string
}
// Close implements resource.BulkResourceWrite.
func (f *folderReader) Close() error {
return nil
}
// CloseWithResults implements resource.BulkResourceWrite.
func (f *folderReader) CloseWithResults() (*resource.BulkResponse, error) {
return &resource.BulkResponse{}, nil
}
// Write implements resource.BulkResourceWrite.
func (f *folderReader) Write(ctx context.Context, key *resource.ResourceKey, value []byte) error {
item := &unstructured.Unstructured{}
err := item.UnmarshalJSON(value)
if err != nil {
return fmt.Errorf("unmarshal unstructured to JSON: %w", err)
}
return f.tree.AddUnstructured(item, f.targetRepoName)
}
func (j *migrationJob) migrateLegacyFolders(ctx context.Context) error {
logger := j.logger
j.progress.SetMessage(ctx, "reading folder tree")
repoName := j.target.Config().Name
j.progress.SetMessage(ctx, "migrate folder tree from legacy")
reader := &folderReader{
tree: j.folderTree,
targetRepoName: repoName,
}
_, err := j.legacy.Migrate(ctx, legacy.MigrateOptions{
Namespace: j.namespace,
Resources: []schema.GroupResource{{
Group: folders.GROUP,
Resource: folders.RESOURCE,
}},
Store: parquet.NewBulkResourceWriterClient(reader),
})
if err != nil {
return fmt.Errorf("unable to read folders from legacy storage %w", err)
}
// create folders first is required so that empty folders exist when finished
j.progress.SetMessage(ctx, "write folders")
err = j.folderTree.Walk(ctx, func(ctx context.Context, folder resources.Folder) error {
p := folder.Path
logger = logger.With("path", p)
result := jobs.JobResourceResult{
Name: folder.ID,
Resource: folders.RESOURCE,
Group: folders.GROUP,
Path: p,
}
_, err := j.target.Read(ctx, p, "")
if err != nil && !(errors.Is(err, repository.ErrFileNotFound) || apierrors.IsNotFound(err)) {
result.Error = fmt.Errorf("failed to check if folder exists before writing: %w", err)
return result.Error
} else if err == nil {
logger.Info("folder already exists")
result.Action = repository.FileActionIgnored
j.progress.Record(ctx, result)
return nil
}
result.Action = repository.FileActionCreated
msg := fmt.Sprintf("export folder %s", p)
// Create with an empty body will make a folder (or .keep file if unsupported)
if err := j.target.Create(ctx, p, "", nil, msg); err != nil {
result.Error = fmt.Errorf("failed to write folder in repo: %w", err)
j.progress.Record(ctx, result)
return result.Error
}
return nil
})
if err != nil {
return fmt.Errorf("failed to write folders: %w", err)
}
return nil
}
pkg/registry/apis/provisioning/jobs/migrate/import.go
+90
View File
@@ -0,0 +1,90 @@
package migrate
import (
"context"
"fmt"
"time"
"google.golang.org/grpc/metadata"
"k8s.io/apimachinery/pkg/runtime/schema"
"github.com/grafana/grafana-app-sdk/logging"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
// called when an error exists
func stopReadingUnifiedStorage(ctx context.Context, dual dualwrite.Service) error {
kinds := []schema.GroupResource{{
Group: folders.GROUP,
Resource: folders.RESOURCE,
}, {
Group: dashboard.GROUP,
Resource: dashboard.DASHBOARD_RESOURCE,
}}
for _, gr := range kinds {
status, _ := dual.Status(ctx, gr)
status.ReadUnified = false
status.Migrated = 0
status.Migrating = 0
_, err := dual.Update(ctx, status)
if err != nil {
return err
}
}
return nil
}
func (j *migrationJob) wipeUnifiedAndSetMigratedFlag(ctx context.Context, dual dualwrite.Service) error {
kinds := []schema.GroupResource{{
Group: folders.GROUP,
Resource: folders.RESOURCE,
}, {
Group: dashboard.GROUP,
Resource: dashboard.DASHBOARD_RESOURCE,
}}
for _, gr := range kinds {
status, _ := dual.Status(ctx, gr)
if status.ReadUnified {
return fmt.Errorf("unexpected state - already using unified storage for: %s", gr)
}
if status.Migrating > 0 {
if time.Since(time.UnixMilli(status.Migrating)) < time.Second*30 {
return fmt.Errorf("another migration job is running for: %s", gr)
}
}
settings := resource.BulkSettings{
RebuildCollection: true, // wipes everything in the collection
Collection: []*resource.ResourceKey{{
Namespace: j.namespace,
Group: gr.Group,
Resource: gr.Resource,
}},
}
ctx = metadata.NewOutgoingContext(ctx, settings.ToMD())
stream, err := j.batch.BulkProcess(ctx)
if err != nil {
return fmt.Errorf("error clearing unified %s / %w", gr, err)
}
stats, err := stream.CloseAndRecv()
if err != nil {
return fmt.Errorf("error clearing unified %s / %w", gr, err)
}
logger := logging.FromContext(ctx)
logger.Error("cleared unified stoage", "stats", stats)
status.Migrated = time.Now().UnixMilli() // but not really... since the sync is starting
status.ReadUnified = true
status.WriteLegacy = false // keep legacy "clean"
_, err = dual.Update(ctx, status)
if err != nil {
return err
}
}
return nil
}
pkg/registry/apis/provisioning/jobs/migrate/resources.go
+234
View File
@@ -0,0 +1,234 @@
package migrate
import (
"context"
"encoding/json"
"fmt"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/client-go/dynamic"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
"github.com/grafana/grafana/pkg/apimachinery/utils"
"github.com/grafana/grafana/pkg/infra/slugify"
"github.com/grafana/grafana/pkg/registry/apis/dashboard/legacy"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
"github.com/grafana/grafana/pkg/storage/unified/parquet"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
var _ resource.BulkResourceWriter = (*resourceReader)(nil)
type resourceReader struct {
job *migrationJob
}
// Close implements resource.BulkResourceWriter.
func (r *resourceReader) Close() error {
return nil
}
// CloseWithResults implements resource.BulkResourceWriter.
func (r *resourceReader) CloseWithResults() (*resource.BulkResponse, error) {
return &resource.BulkResponse{}, nil
}
// Write implements resource.BulkResourceWriter.
func (r *resourceReader) Write(ctx context.Context, key *resource.ResourceKey, value []byte) error {
// Reuse the same parse+cleanup logic
parsed, err := r.job.parser.Parse(ctx, &repository.FileInfo{
Path: "", // empty path to ignore file system
Data: value,
}, false)
if err != nil {
// TODO: should we fail the entire execution?
return fmt.Errorf("failed to unmarshal unstructured: %w", err)
}
// clear anything so it will get written
parsed.Meta.SetManagerProperties(utils.ManagerProperties{})
parsed.Meta.SetSourceProperties(utils.SourceProperties{})
if result := r.job.write(ctx, parsed.Obj); result.Error != nil {
r.job.progress.Record(ctx, result)
if err := r.job.progress.TooManyErrors(); err != nil {
return err
}
}
return nil
}
func (j *migrationJob) migrateLegacyResources(ctx context.Context) error {
kinds := []schema.GroupVersionResource{{
Group: dashboard.GROUP,
Resource: dashboard.DASHBOARD_RESOURCE,
Version: "v1alpha1",
}}
for _, kind := range kinds {
j.progress.SetMessage(ctx, fmt.Sprintf("migrate %s resource", kind.Resource))
gr := kind.GroupResource()
opts := legacy.MigrateOptions{
Namespace: j.namespace,
WithHistory: j.options.History,
Resources: []schema.GroupResource{gr},
Store: parquet.NewBulkResourceWriterClient(&resourceReader{job: j}),
OnlyCount: true, // first get the count
}
stats, err := j.legacy.Migrate(ctx, opts)
if err != nil {
return fmt.Errorf("unable to count legacy items %w", err)
}
// FIXME: explain why we calculate it in this way
if len(stats.Summary) > 0 {
count := stats.Summary[0].Count //
history := stats.Summary[0].History
if history > count {
count = history // the number of items we will process
}
j.progress.SetTotal(ctx, int(count))
}
opts.OnlyCount = false // this time actually write
_, err = j.legacy.Migrate(ctx, opts)
if err != nil {
return fmt.Errorf("error running legacy migrate %s %w", kind.Resource, err)
}
}
return nil
}
func (j *migrationJob) write(ctx context.Context, obj *unstructured.Unstructured) jobs.JobResourceResult {
gvk := obj.GroupVersionKind()
result := jobs.JobResourceResult{
Name: obj.GetName(),
Resource: gvk.Kind,
Group: gvk.Group,
Action: repository.FileActionCreated,
}
if err := ctx.Err(); err != nil {
result.Error = fmt.Errorf("context error: %w", err)
return result
}
meta, err := utils.MetaAccessor(obj)
if err != nil {
result.Error = fmt.Errorf("extract meta accessor: %w", err)
return result
}
// Message from annotations
commitMessage := meta.GetMessage()
if commitMessage == "" {
g := meta.GetGeneration()
if g > 0 {
commitMessage = fmt.Sprintf("Generation: %d", g)
} else {
commitMessage = "exported from grafana"
}
}
name := meta.GetName()
manager, _ := meta.GetManagerProperties()
if manager.Identity == j.target.Config().GetName() {
result.Action = repository.FileActionIgnored
return result
}
title := meta.FindTitle("")
if title == "" {
title = name
}
folder := meta.GetFolder()
// Add the author in context (if available)
ctx = j.withAuthorSignature(ctx, meta)
// Get the absolute path of the folder
fid, ok := j.folderTree.DirPath(folder, "")
if !ok {
// FIXME: Shouldn't this fail instead?
fid = resources.Folder{
Path: "__folder_not_found/" + slugify.Slugify(folder),
}
j.logger.Error("folder of item was not in tree of repository")
}
result.Path = fid.Path
// Clear the metadata
delete(obj.Object, "metadata")
if j.options.Identifier {
meta.SetName(name) // keep the identifier in the metadata
}
body, err := json.MarshalIndent(obj.Object, "", " ")
if err != nil {
result.Error = fmt.Errorf("failed to marshal dashboard: %w", err)
return result
}
fileName := slugify.Slugify(title) + ".json"
if fid.Path != "" {
fileName = safepath.Join(fid.Path, fileName)
}
err = j.target.Write(ctx, fileName, "", body, commitMessage)
if err != nil {
result.Error = fmt.Errorf("failed to write file: %w", err)
}
return result
}
func removeUnprovisioned(ctx context.Context, client dynamic.ResourceInterface, progress jobs.JobProgressRecorder) error {
rawList, err := client.List(ctx, metav1.ListOptions{Limit: 10000})
if err != nil {
return fmt.Errorf("failed to list resources: %w", err)
}
if rawList.GetContinue() != "" {
return fmt.Errorf("unable to list all resources in one request: %s", rawList.GetContinue())
}
for _, item := range rawList.Items {
// Create a pointer to the item since MetaAccessor requires a pointer
itemPtr := &item
meta, err := utils.MetaAccessor(itemPtr)
if err != nil {
return fmt.Errorf("extract meta accessor: %w", err)
}
// Skip if managed
_, ok := meta.GetManagerProperties()
if ok {
continue
}
result := jobs.JobResourceResult{
Name: item.GetName(),
Resource: item.GetKind(),
Group: item.GroupVersionKind().Group,
Action: repository.FileActionDeleted,
}
if err = client.Delete(ctx, item.GetName(), metav1.DeleteOptions{}); err != nil {
result.Error = fmt.Errorf("failed to delete folder: %w", err)
progress.Record(ctx, result)
return result.Error
}
progress.Record(ctx, result)
}
return nil
}
pkg/registry/apis/provisioning/jobs/migrate/users.go
+53
View File
@@ -0,0 +1,53 @@
package migrate
import (
"context"
"fmt"
"strings"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
func (j *migrationJob) loadUsers(ctx context.Context) error {
client, err := j.parser.Clients().User()
if err != nil {
return err
}
rawList, err := client.List(ctx, metav1.ListOptions{Limit: 10000})
if err != nil {
return fmt.Errorf("failed to list users: %w", err)
}
if rawList.GetContinue() != "" {
return fmt.Errorf("unable to list all users in one request: %s", rawList.GetContinue())
}
var ok bool
j.userInfo = make(map[string]repository.CommitSignature)
for _, item := range rawList.Items {
sig := repository.CommitSignature{}
// FIXME: should we improve logging here?
sig.Name, ok, err = unstructured.NestedString(item.Object, "spec", "login")
if !ok || err != nil {
continue
}
sig.Email, ok, err = unstructured.NestedString(item.Object, "spec", "email")
if !ok || err != nil {
continue
}
if sig.Name == sig.Email {
if sig.Name == "" {
sig.Name = item.GetName()
} else if strings.Contains(sig.Email, "@") {
sig.Email = "" // don't use the same value for name+email
}
}
j.userInfo["user:"+item.GetName()] = sig
}
return nil
}
pkg/registry/apis/provisioning/jobs/migrate/worker.go
+333
View File
@@ -0,0 +1,333 @@
package migrate
import (
"bufio"
"context"
"errors"
"fmt"
"io"
"time"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/utils"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/dashboard/legacy"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs/export"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs/sync"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
gogit "github.com/grafana/grafana/pkg/registry/apis/provisioning/repository/go-git"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/secrets"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
type MigrationWorker struct {
// Tempdir for repo clones
clonedir string
// temporary... while we still do an import
parsers *resources.ParserFactory
// Check where values are currently saved
storageStatus dualwrite.Service
// Support reading from history
legacyMigrator legacy.LegacyMigrator
// Direct access to unified storage... use carefully!
bulk resource.BulkStoreClient
// Decrypt secret from config object
secrets secrets.Service
// Delegate the export to the export worker
exportWorker *export.ExportWorker
// Delegate the import to sync worker
syncWorker *sync.SyncWorker
}
func NewMigrationWorker(
legacyMigrator legacy.LegacyMigrator,
parsers *resources.ParserFactory, // should not be necessary!
storageStatus dualwrite.Service,
batch resource.BulkStoreClient,
secrets secrets.Service,
exportWorker *export.ExportWorker,
syncWorker *sync.SyncWorker,
clonedir string,
) *MigrationWorker {
return &MigrationWorker{
clonedir,
parsers,
storageStatus,
legacyMigrator,
batch,
secrets,
exportWorker,
syncWorker,
}
}
func (w *MigrationWorker) IsSupported(ctx context.Context, job provisioning.Job) bool {
return job.Spec.Action == provisioning.JobActionMigrate
}
// Process will start a job
func (w *MigrationWorker) Process(ctx context.Context, repo repository.Repository, job provisioning.Job, progress jobs.JobProgressRecorder) error {
options := job.Spec.Migrate
if options == nil {
return errors.New("missing migrate settings")
}
var (
err error
buffered *gogit.GoGitRepo
)
isFromLegacy := dualwrite.IsReadingLegacyDashboardsAndFolders(ctx, w.storageStatus)
progress.SetTotal(ctx, 10) // will show a progress bar
// TODO: we should fail fast if migration is not possible and not always clone the repository.
if repo.Config().Spec.GitHub != nil {
progress.SetMessage(ctx, "clone "+repo.Config().Spec.GitHub.URL)
reader, writer := io.Pipe()
go func() {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
progress.SetMessage(ctx, scanner.Text())
}
}()
buffered, err = gogit.Clone(ctx, repo.Config(), gogit.GoGitCloneOptions{
Root: w.clonedir,
SingleCommitBeforePush: !(options.History && isFromLegacy),
// TODO: make this configurable
Timeout: 10 * time.Minute,
}, w.secrets, writer)
if err != nil {
return fmt.Errorf("unable to clone target: %w", err)
}
repo = buffered // send all writes to the buffered repo
defer func() {
if err := buffered.Remove(ctx); err != nil {
logging.FromContext(ctx).Error("failed to remove cloned repository after migrate", "err", err)
}
}()
}
rw, ok := repo.(repository.ReaderWriter)
if !ok {
return errors.New("migration job submitted targeting repository that is not a ReaderWriter")
}
if isFromLegacy {
return w.migrateFromLegacy(ctx, rw, buffered, *options, progress)
}
return w.migrateFromUnifiedStorage(ctx, rw, *options, progress)
}
// migrateFromLegacy will export the resources from legacy storage and import them into the target repository
func (w *MigrationWorker) migrateFromLegacy(ctx context.Context, rw repository.ReaderWriter, buffered *gogit.GoGitRepo, options provisioning.MigrateJobOptions, progress jobs.JobProgressRecorder) error {
parser, err := w.parsers.GetParser(ctx, rw)
if err != nil {
return fmt.Errorf("error getting parser: %w", err)
}
worker, err := newMigrationJob(ctx, rw, options, parser, w.bulk, w.legacyMigrator, progress)
if err != nil {
return fmt.Errorf("error creating job: %w", err)
}
if options.History {
progress.SetMessage(ctx, "loading users")
err = worker.loadUsers(ctx)
if err != nil {
return fmt.Errorf("error loading users: %w", err)
}
}
progress.SetMessage(ctx, "exporting legacy folders")
err = worker.migrateLegacyFolders(ctx)
if err != nil {
return err
}
progress.SetMessage(ctx, "exporting legacy resources")
err = worker.migrateLegacyResources(ctx)
if err != nil {
return err
}
if buffered != nil {
progress.SetMessage(ctx, "pushing changes")
reader, writer := io.Pipe()
go func() {
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
progress.SetMessage(ctx, scanner.Text())
}
}()
if err := buffered.Push(ctx, gogit.GoGitPushOptions{
// TODO: make this configurable
Timeout: 10 * time.Minute,
}, writer); err != nil {
return fmt.Errorf("error pushing changes: %w", err)
}
}
progress.SetMessage(ctx, "resetting unified storage")
if err = worker.wipeUnifiedAndSetMigratedFlag(ctx, w.storageStatus); err != nil {
return fmt.Errorf("unable to reset unified storage %w", err)
}
// Reset the results after the export as pull will operate on the same resources
progress.ResetResults()
// Delegate the import to a sync (from the already checked out go-git repository!)
progress.SetMessage(ctx, "pulling resources")
err = w.syncWorker.Process(ctx, rw, provisioning.Job{
Spec: provisioning.JobSpec{
Pull: &provisioning.SyncJobOptions{
Incremental: false,
},
},
}, progress)
if err != nil { // this will have an error when too many errors exist
progress.SetMessage(ctx, "error importing resources, reverting")
if e2 := stopReadingUnifiedStorage(ctx, w.storageStatus); e2 != nil {
logger := logging.FromContext(ctx)
logger.Warn("error trying to revert dual write settings after an error", "err", err)
}
}
return err
}
// migrateFromUnifiedStorage will export the resources from unified storage and import them into the target repository
func (w *MigrationWorker) migrateFromUnifiedStorage(ctx context.Context, repo repository.ReaderWriter, options provisioning.MigrateJobOptions, progress jobs.JobProgressRecorder) error {
parser, err := w.parsers.GetParser(ctx, repo)
if err != nil {
return fmt.Errorf("error getting parser: %w", err)
}
progress.SetMessage(ctx, "exporting unified storage resources")
if err := w.exportWorker.Process(ctx, repo, provisioning.Job{
Spec: provisioning.JobSpec{
Push: &provisioning.ExportJobOptions{
Identifier: options.Identifier,
},
},
}, progress); err != nil {
return fmt.Errorf("export resources: %w", err)
}
// Reset the results after the export as pull will operate on the same resources
progress.ResetResults()
progress.SetMessage(ctx, "pulling resources")
err = w.syncWorker.Process(ctx, repo, provisioning.Job{
Spec: provisioning.JobSpec{
Pull: &provisioning.SyncJobOptions{
Incremental: false,
},
},
}, progress)
if err != nil {
return fmt.Errorf("pull resources: %w", err)
}
folderClient, err := parser.Clients().Folder()
if err != nil {
return fmt.Errorf("unable to get folder client: %w", err)
}
dashboardClient, err := parser.Clients().Dashboard()
if err != nil {
return fmt.Errorf("unable to get dashboard client: %w", err)
}
progress.SetMessage(ctx, "removing unprovisioned folders")
err = removeUnprovisioned(ctx, folderClient, progress)
if err != nil {
return fmt.Errorf("remove unprovisioned folders: %w", err)
}
progress.SetMessage(ctx, "removing unprovisioned dashboards")
err = removeUnprovisioned(ctx, dashboardClient, progress)
if err != nil {
return fmt.Errorf("remove unprovisioned dashboards: %w", err)
}
return nil
}
// MigrationJob holds all context for a running job
type migrationJob struct {
logger logging.Logger
target repository.ReaderWriter
legacy legacy.LegacyMigrator
parser *resources.Parser
batch resource.BulkStoreClient
namespace string
progress jobs.JobProgressRecorder
userInfo map[string]repository.CommitSignature
folderTree *resources.FolderTree
options provisioning.MigrateJobOptions
}
func newMigrationJob(ctx context.Context,
target repository.ReaderWriter,
options provisioning.MigrateJobOptions,
parser *resources.Parser,
batch resource.BulkStoreClient,
legacyMigrator legacy.LegacyMigrator,
progress jobs.JobProgressRecorder,
) (*migrationJob, error) {
return &migrationJob{
namespace: target.Config().Namespace,
target: target,
logger: logging.FromContext(ctx),
progress: progress,
options: options,
parser: parser,
batch: batch,
legacy: legacyMigrator,
folderTree: resources.NewEmptyFolderTree(),
}, nil
}
func (j *migrationJob) withAuthorSignature(ctx context.Context, item utils.GrafanaMetaAccessor) context.Context {
if j.userInfo == nil {
return ctx
}
id := item.GetUpdatedBy()
if id == "" {
id = item.GetCreatedBy()
}
if id == "" {
id = "grafana"
}
sig := j.userInfo[id] // lookup
if sig.Name == "" && sig.Email == "" {
sig.Name = id
}
t, err := item.GetUpdatedTimestamp()
if err == nil && t != nil {
sig.When = *t
} else {
sig.When = item.GetCreationTimestamp().Time
}
return repository.WithAuthorSignature(ctx, sig)
}
pkg/registry/apis/provisioning/jobs/persistentstore.go
+427
View File
@@ -0,0 +1,427 @@
package jobs
import (
"context"
"errors"
"fmt"
"net/http"
"strconv"
"time"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/identity"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/apifmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/internalversion"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/selection"
"k8s.io/apiserver/pkg/endpoints/request"
"k8s.io/apiserver/pkg/registry/rest"
)
const (
// LabelJobClaim includes the timestamp when the job was claimed.
// The label must be formatted as milliseconds from Epoch. This grants a natural ordering, allowing for less-than operators in label selectors.
// The natural ordering would be broken if the number rolls over into 1 more digit. This won't happen before Nov, 2286.
LabelJobClaim = "provisioning.grafana.app/claim"
// LabelJobOriginalName contains the Job's name as a label. This allows for label selectors to find the archived version of a job.
LabelJobOriginalName = "provisioning.grafana.app/original-name"
// LabelRepository contains the repository name as a label. This allows for label selectors to find the archived version of a job.
LabelRepository = "provisioning.grafana.app/repository"
)
var (
ErrNoJobs = &apierrors.StatusError{
ErrStatus: metav1.Status{
Status: metav1.StatusFailure,
Reason: metav1.StatusReasonConflict,
Message: "no jobs are available to claim, try again later",
Code: http.StatusNoContent,
Details: &metav1.StatusDetails{
Group: provisioning.GROUP,
Kind: provisioning.JobResourceInfo.GetName(),
RetryAfterSeconds: 3,
},
},
}
errWouldCreate = errors.New("this call would have created a new resource; it is rejected")
failCreation rest.ValidateObjectFunc = func(_ context.Context, _ runtime.Object) error {
return errWouldCreate
}
)
type Queue interface {
// Insert adds a new job to the queue.
//
// This saves it if it is a new job, or fails with ErrJobAlreadyExists if one with the same name already exists.
Insert(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error)
}
var _ Queue = (*persistentStore)(nil)
type jobStorage interface {
rest.Creater
rest.Lister
rest.Patcher
rest.GracefulDeleter
}
// persistentStore is a job queue abstraction.
// It calls out to a real storage implementation to store the jobs, and a separate storage for historic jobs that have been completed.
// When persistentStore claims a job, it will update the status of it. This does a ResourceVersion check to ensure it is atomic; if the job has been claimed by another worker, the claim will fail.
// When a job is completed, it is moved to the historic job store by first deleting it from the job store and then creating it in the historic job store. We are fine with the job being lost if the historic job store fails to create it.
type persistentStore struct {
jobStore jobStorage
historicJobStore rest.Creater
// clock is a function that returns the current time.
clock func() time.Time
// expiry is the time after which a job is considered abandoned.
// If a job is abandoned, it will have its claim cleaned up periodically.
expiry time.Duration
// notifications has a signal sent to it when a new job is inserted. If a value already exists, nothing is sent.
//
// This is very similar to the concept of a Waker in Rust: <https://doc.rust-lang.org/std/task/struct.Waker.html>
notifications chan struct{}
}
func NewStore(
jobStore jobStorage,
historicJobStore rest.Creater,
expiry time.Duration,
) (*persistentStore, error) {
if expiry <= 0 {
expiry = time.Second * 30
}
return &persistentStore{
jobStore: jobStore,
historicJobStore: historicJobStore,
clock: time.Now,
expiry: expiry,
notifications: make(chan struct{}, 1),
}, nil
}
// Claim takes a job from storage, marks it as ours, and returns it.
//
// Any job which has not been claimed by another worker is fair game.
//
// If err is not nil, the job and rollback values are always nil.
// The err may be ErrNoJobs if there are no jobs to claim.
func (s *persistentStore) Claim(ctx context.Context) (job *provisioning.Job, rollback func(), err error) {
requirement, err := labels.NewRequirement(LabelJobClaim, selection.DoesNotExist, nil)
if err != nil {
return nil, nil, apifmt.Errorf("could not create requirement: %w", err)
}
jobsObj, err := s.jobStore.List(ctx, &internalversion.ListOptions{
LabelSelector: labels.NewSelector().Add(*requirement),
Limit: 16,
})
if err != nil {
return nil, nil, apifmt.Errorf("failed to list jobs: %w", err)
}
jobs, ok := jobsObj.(*provisioning.JobList)
if !ok {
return nil, nil, apifmt.Errorf("unexpected object type %T", jobsObj)
}
if len(jobs.Items) == 0 {
return nil, nil, ErrNoJobs
}
for _, job := range jobs.Items {
if job.Labels == nil {
job.Labels = make(map[string]string)
}
job.Labels[LabelJobClaim] = strconv.FormatInt(s.clock().UnixMilli(), 10)
// We list jobs from all namespaces. So when we want to update a specific job, we also need its namespace in the context.
ctx := request.WithNamespace(ctx, job.GetNamespace())
// Likewise, we should use the provisioning identity now that we have the namespace we are operating within.
ctx, _, err = identity.WithProvisioningIdentity(ctx, job.GetNamespace())
if err != nil {
// This should never happen, as it is already a valid namespace from the job existing... but better be safe.
return nil, nil, apifmt.Errorf("failed to get provisioning identity for '%s': %w", job.GetNamespace(), err)
}
// This relies on the resource version being updated for us.
// If the resource version we pass in via the current job is not the same as the one currently in the store, it will fail with Conflict.
// This is the desired behavior, as it ensures that claims are atomic.
updated, _, err := s.jobStore.Update(ctx,
job.GetName(), // name
rest.DefaultUpdatedObjectInfo(&job), // objInfo
failCreation, // createValidation
nil, // updateValidation
false, // forceAllowCreate
&metav1.UpdateOptions{}, // options
)
if apierrors.IsConflict(err) || errors.Is(err, errWouldCreate) {
// On conflict: another worker claimed the job before us.
// On would create: the job was completed and deleted before we could claim it.
// We'll just move on to the next job.
continue
}
if err != nil {
return nil, nil, apifmt.Errorf("failed to claim job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
updatedJob, ok := updated.(*provisioning.Job)
if !ok {
return nil, nil, apifmt.Errorf("unexpected object type %T", updated)
}
return updatedJob.DeepCopy(), func() {
// Rolling back does not need to care about the parent's cancellation state.
// This will also use the parent context (i.e. from the for loop!), ensuring we have permissions to do this.
ctx = context.WithoutCancel(ctx)
logger := logging.FromContext(ctx).With("namespace", updatedJob.GetNamespace(), "job", updatedJob.GetName())
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
refetched, err := s.jobStore.Get(timeoutCtx, updatedJob.GetName(), &metav1.GetOptions{})
cancel() // we have no response body to read (the obj already contains all of it), so just cancel immediately
if apierrors.IsNotFound(err) {
// The job was probably completed already. Nothing to roll back!
return
} else if err != nil {
// We failed. Nothing much we can do but let the job be cleaned up by the periodic cleaner.
logger.Warn("failed to roll back job claim; letting periodic cleaner deal with it", "error", err)
return
}
refetchedJob, ok := refetched.(*provisioning.Job)
if !ok {
logger.Warn("failed to roll back job claim: the job we got is not a *provisioning.Job?", "got", refetched)
return
}
// Rollback the claim.
delete(refetchedJob.Labels, LabelJobClaim)
refetchedJob.Status.State = provisioning.JobStatePending
timeoutCtx, cancel = context.WithTimeout(ctx, 5*time.Second)
_, _, err = s.jobStore.Update(timeoutCtx,
refetchedJob.GetName(), // name
rest.DefaultUpdatedObjectInfo(refetchedJob), // objInfo
failCreation, // createValidation
nil, // updateValidation
false, // forceAllowCreate
&metav1.UpdateOptions{}, // options
)
cancel() // we have no response body to read (the obj already contains all of it), so just cancel immediately
if err != nil && !apierrors.IsConflict(err) && !errors.Is(err, errWouldCreate) {
logger.Warn("failed to roll back job claim; letting periodic cleaner deal with it", "error", err)
} else if err != nil {
logger.Debug("failed to roll back job claim; got an OK error", "error", err)
}
}, nil
}
// We failed to claim any jobs.
return nil, nil, ErrNoJobs
}
// Update saves the job back to the store.
func (s *persistentStore) Update(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error) {
obj, _, err := s.jobStore.Update(ctx,
job.GetName(), // name
rest.DefaultUpdatedObjectInfo(job), // objInfo
failCreation, // createValidation
nil, // updateValidation
false, // forceAllowCreate
&metav1.UpdateOptions{}, // options
)
if err != nil {
return nil, apifmt.Errorf("failed to update job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
updatedJob, ok := obj.(*provisioning.Job)
if !ok {
return nil, apifmt.Errorf("unexpected object type %T", obj)
}
return updatedJob, nil
}
// Complete marks a job as completed and moves it to the historic job store.
// When in the historic store, there is no more claim on the job.
func (s *persistentStore) Complete(ctx context.Context, job *provisioning.Job) error {
logger := logging.FromContext(ctx).With("namespace", job.GetNamespace(), "job", job.GetName())
// We need to delete the job from the job store and create it in the historic job store.
// We are fine with the job being lost if the historic job store fails to create it.
//
// We will assume that the caller is the claimant. If this is not true, an error is returned.
// This is a best-effort operation; if the job is not in the claimed state, we will still attempt to move it to the historic job store.
_, _, err := s.jobStore.Delete(ctx, job.GetName(), nil, &metav1.DeleteOptions{})
if err != nil {
return apifmt.Errorf("failed to delete job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
logger.Debug("deleted job from job store")
// We need to remove the claim label before moving the job to the historic job store.
if job.Labels == nil {
job.Labels = make(map[string]string)
}
delete(job.Labels, LabelJobClaim)
// We also need a new, unique name.
job.Labels[LabelJobOriginalName] = job.GetName()
job.Labels[LabelRepository] = job.Spec.Repository
job.GenerateName = job.Name + "-"
job.Name = ""
// We also reset the UID as this is not the same object.
job.ObjectMeta.UID = ""
// We aren't allowed to write with ResourceVersion set.
job.ResourceVersion = ""
historicJob := &provisioning.HistoricJob{
ObjectMeta: job.ObjectMeta,
Spec: job.Spec,
Status: job.Status,
}
_, err = s.historicJobStore.Create(ctx, historicJob, nil, &metav1.CreateOptions{})
if err != nil {
// We're not going to return this as it is not critical. Not ideal, but not critical.
logger.Warn("failed to create historic job", "historic_job", *historicJob, "error", err)
} else {
logger.Debug("created historic job", "historic_job", *historicJob)
}
logger.Debug("job completion done")
return nil
}
// Cleanup should be called periodically to clean up abandoned jobs.
// An abandoned job is one that has been claimed by a worker, but the worker has not updated the job in a while.
func (s *persistentStore) Cleanup(ctx context.Context) error {
if err := s.cleanupClaims(ctx); err != nil {
return apifmt.Errorf("failed to clean up claims: %w", err)
}
return nil
}
// cleanupClaims will clean up abandoned claims.
// Any claim that is older than the expiry time will have their claims removed.
//
// This is only necessary because Kubernetes does not support logical OR in label selectors.
func (s *persistentStore) cleanupClaims(ctx context.Context) error {
// We will list all jobs that have been claimed but not updated in a while.
// We will then remove the claim from them.
// We will not care about the result of the update, as the job may have been completed in the meantime.
expiry := s.clock().Add(-s.expiry).UnixMilli()
requirement, err := labels.NewRequirement(LabelJobClaim, selection.LessThan, []string{strconv.FormatInt(expiry, 10)})
if err != nil {
return apifmt.Errorf("could not create requirement: %w", err)
}
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
jobsObj, err := s.jobStore.List(timeoutCtx, &internalversion.ListOptions{
LabelSelector: labels.NewSelector().Add(*requirement),
// We don't need to clean up everything all the time. Just do enough such that we have a fair amount of work.
Limit: 100,
})
cancel() // by the time we have the list, there is no response body to read, so just cancel immediately
if err != nil {
return apifmt.Errorf("failed to list jobs: %w", err)
}
jobs, ok := jobsObj.(*provisioning.JobList)
if !ok {
return apifmt.Errorf("unexpected object type %T", jobsObj)
}
for _, job := range jobs.Items {
if job.Labels == nil {
job.Labels = make(map[string]string)
}
delete(job.Labels, LabelJobClaim)
job.Status.State = provisioning.JobStatePending
// We list jobs from all namespaces. So when we want to update a specific job, we also need its namespace in the context.
ctx := request.WithNamespace(ctx, job.GetNamespace())
// Likewise, we should use the provisioning identity now that we have the namespace we are operating within.
ctx, _, err = identity.WithProvisioningIdentity(ctx, job.GetNamespace())
if err != nil {
// This should never happen, as it is already a valid namespace from the job existing... but better be safe.
return apifmt.Errorf("failed to get provisioning identity for '%s': %w", job.GetNamespace(), err)
}
timeoutCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
_, _, err := s.jobStore.Update(timeoutCtx,
job.GetName(), // name
rest.DefaultUpdatedObjectInfo(&job), // objInfo
failCreation, // createValidation
nil, // updateValidation
false, // forceAllowCreate
&metav1.UpdateOptions{}, // options
)
cancel() // we have no response body to read, so just cancel immediately
if apierrors.IsConflict(err) || errors.Is(err, errWouldCreate) {
continue
}
if err != nil {
return apifmt.Errorf("failed to unclaim job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
}
return nil
}
// Insert adds a new job to the queue.
//
// The job name is not honoured. It will be overwritten with a name that fits the job.
//
// This saves it if it is a new job, or fails with `apierrors.IsAlreadyExists(err) == true` if one already exists.
func (s *persistentStore) Insert(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error) {
s.generateJobName(job) // Side-effect: updates the job's name.
obj, err := s.jobStore.Create(ctx, job, nil, &metav1.CreateOptions{})
if apierrors.IsAlreadyExists(err) {
return nil, apifmt.Errorf("job '%s' in '%s' already exists: %w", job.GetName(), job.GetNamespace(), err)
}
if err != nil {
return nil, apifmt.Errorf("failed to create job '%s' in '%s': %w", job.GetName(), job.GetNamespace(), err)
}
created, ok := obj.(*provisioning.Job)
if !ok {
return nil, apifmt.Errorf("unexpected object type %T", obj)
}
select {
case s.notifications <- struct{}{}:
default:
// We don't want to block if there is already a notification waiting.
}
return created, nil
}
func (s *persistentStore) InsertNotifications() chan struct{} {
return s.notifications
}
// generateJobName creates and updates the job's name to one that fits it.
func (s *persistentStore) generateJobName(job *provisioning.Job) {
switch job.Spec.Action {
case provisioning.JobActionMigrate, provisioning.JobActionSync:
// Sync and migrate jobs should never run at the same time. Hence, the name encapsulates them both (and the spec differentiates them).
job.Name = job.Spec.Repository + "-syncmigrate"
case provisioning.JobActionPullRequest:
var pr int
if job.Spec.PullRequest != nil {
pr = job.Spec.PullRequest.PR
}
// There may be multiple pull requests at the same time. They need different names.
job.Name = fmt.Sprintf("%s-pr-%d", job.Spec.Repository, pr)
default:
job.Name = fmt.Sprintf("%s-%s", job.Spec.Repository, job.Spec.Action)
}
}
pkg/registry/apis/provisioning/jobs/progress.go
+53 -21
View File
@@ -36,22 +36,26 @@ type JobResourceResult struct {
}
type jobProgressRecorder struct {
started time.Time
total int
ref string
message string
resultCount int
errorCount int
errors []string
progressFn ProgressFn
summaries map[string]*provisioning.JobResourceSummary
started time.Time
total int
ref string
message string
finalMessage string
resultCount int
errorCount int
errors []string
notifyImmediatelyFn ProgressFn
maybeNotifyFn ProgressFn
summaries map[string]*provisioning.JobResourceSummary
}
func newJobProgressRecorder(ProgressFn ProgressFn) JobProgressRecorder {
return &jobProgressRecorder{
started: time.Now(),
progressFn: maybeNotifyProgress(5*time.Second, ProgressFn),
summaries: make(map[string]*provisioning.JobResourceSummary),
started: time.Now(),
// Have a faster notifier for messages and total
notifyImmediatelyFn: maybeNotifyProgress(500*time.Millisecond, ProgressFn),
maybeNotifyFn: maybeNotifyProgress(5*time.Second, ProgressFn),
summaries: make(map[string]*provisioning.JobResourceSummary),
}
}
@@ -70,15 +74,26 @@ func (r *jobProgressRecorder) Record(ctx context.Context, result JobResourceResu
}
r.updateSummary(result)
r.notify(ctx)
r.maybeNotify(ctx)
}
func (r *jobProgressRecorder) SetMessage(msg string) {
// ResetResults will reset the results of the job
func (r *jobProgressRecorder) ResetResults() {
r.resultCount = 0
r.errorCount = 0
r.errors = nil
r.summaries = make(map[string]*provisioning.JobResourceSummary)
}
func (r *jobProgressRecorder) SetMessage(ctx context.Context, msg string) {
r.message = msg
logging.FromContext(ctx).Info("job progress message", "message", msg)
r.notifyImmediately(ctx)
}
func (r *jobProgressRecorder) GetMessage() string {
return r.message
func (r *jobProgressRecorder) SetFinalMessage(ctx context.Context, msg string) {
r.finalMessage = msg
logging.FromContext(ctx).Info("job final message", "message", msg)
}
func (r *jobProgressRecorder) SetRef(ref string) {
@@ -89,8 +104,10 @@ func (r *jobProgressRecorder) GetRef() string {
return r.ref
}
func (r *jobProgressRecorder) SetTotal(total int) {
func (r *jobProgressRecorder) SetTotal(ctx context.Context, total int) {
r.total = total
r.notifyImmediately(ctx)
}
func (r *jobProgressRecorder) TooManyErrors() error {
@@ -154,7 +171,7 @@ func (r *jobProgressRecorder) progress() float64 {
return float64(r.resultCount) / float64(r.total) * 100
}
func (r *jobProgressRecorder) notify(ctx context.Context) {
func (r *jobProgressRecorder) notifyImmediately(ctx context.Context) {
jobStatus := provisioning.JobStatus{
State: provisioning.JobStateWorking,
Message: r.message,
@@ -164,7 +181,22 @@ func (r *jobProgressRecorder) notify(ctx context.Context) {
}
logger := logging.FromContext(ctx)
if err := r.progressFn(ctx, jobStatus); err != nil {
if err := r.notifyImmediatelyFn(ctx, jobStatus); err != nil {
logger.Warn("error notifying immediate progress", "err", err)
}
}
func (r *jobProgressRecorder) maybeNotify(ctx context.Context) {
jobStatus := provisioning.JobStatus{
State: provisioning.JobStateWorking,
Message: r.message,
Errors: r.errors,
Progress: r.progress(),
Summary: r.summary(),
}
logger := logging.FromContext(ctx)
if err := r.maybeNotifyFn(ctx, jobStatus); err != nil {
logger.Warn("error notifying progress", "err", err)
}
}
@@ -195,8 +227,8 @@ func (r *jobProgressRecorder) Complete(ctx context.Context, err error) provision
}
// Override message if progress have a more explicit message
if r.message != "" && jobStatus.State != provisioning.JobStateError {
jobStatus.Message = r.message
if r.finalMessage != "" && jobStatus.State != provisioning.JobStateError {
jobStatus.Message = r.finalMessage
}
return jobStatus
pkg/registry/apis/provisioning/jobs/pullrequest/preview.go
+158
View File
@@ -0,0 +1,158 @@
package pullrequest
import (
"bytes"
"context"
"fmt"
"html/template"
"net/url"
"path"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/cmd/grafana-cli/logger"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
// resourcePreview represents a resource that has changed in a pull request.
type resourcePreview struct {
Filename string
Path string
Action string
Kind string
OriginalURL string
PreviewURL string
PreviewScreenshotURL string
}
const previewsCommentTemplate = `Hey there! 🎉
Grafana spotted some changes for your resources in this pull request:
## Summary
| File Name | Kind | Path | Action | Links |
|-----------|------|------|--------|-------|
{{- range .}}
| {{.Filename}} | {{.Kind}} | {{.Path}} | {{.Action}} | {{- if .OriginalURL}}[Original]({{.OriginalURL}}){{- end}}{{- if and .OriginalURL .PreviewURL}}, {{end}}{{- if .PreviewURL}}[Preview]({{.PreviewURL}}){{- end}} |
{{- end}}
Click the preview links above to view how your changes will look and compare them with the original and current versions.
{{- range .}}
{{- if .PreviewScreenshotURL}}
### Preview of {{.Filename}}
![Preview]({{.PreviewScreenshotURL}})
{{- end}}{{- end}}`
// PreviewRenderer is an interface for rendering a preview of a file
type PreviewRenderer interface {
IsAvailable(ctx context.Context) bool
RenderDashboardPreview(ctx context.Context, namespace, repoName, path, ref string) (string, error)
}
type Previewer struct {
template *template.Template
urlProvider func(namespace string) string
renderer PreviewRenderer
}
func NewPreviewer(renderer PreviewRenderer, urlProvider func(namespace string) string) *Previewer {
return &Previewer{
template: template.Must(template.New("comment").Parse(previewsCommentTemplate)),
urlProvider: urlProvider,
renderer: renderer,
}
}
// GenerateComment creates a formatted comment for dashboard previews
func (p *Previewer) GenerateComment(previews []resourcePreview) (string, error) {
var buf bytes.Buffer
if err := p.template.Execute(&buf, previews); err != nil {
return "", fmt.Errorf("execute previews comment template: %w", err)
}
return buf.String(), nil
}
// getOriginalURL returns the URL for the original version of the file based on the action
func (p *Previewer) getOriginalURL(ctx context.Context, f repository.VersionedFileChange, baseURL *url.URL, repoName, base, pullRequestURL string) string {
switch f.Action {
case repository.FileActionCreated:
return "" // No original URL for new files
case repository.FileActionUpdated:
return p.previewURL(baseURL, repoName, base, f.Path, pullRequestURL)
case repository.FileActionRenamed:
return p.previewURL(baseURL, repoName, base, f.PreviousPath, pullRequestURL)
case repository.FileActionDeleted:
return p.previewURL(baseURL, repoName, base, f.Path, pullRequestURL)
default:
logging.FromContext(ctx).Error("unknown file action for original URL", "action", f.Action)
return ""
}
}
// getPreviewURL returns the URL for the preview version of the file based on the action
func (p *Previewer) getPreviewURL(ctx context.Context, f repository.VersionedFileChange, baseURL *url.URL, repoName, ref, pullRequestURL string) string {
switch f.Action {
case repository.FileActionCreated, repository.FileActionUpdated, repository.FileActionRenamed:
return p.previewURL(baseURL, repoName, ref, f.Path, pullRequestURL)
case repository.FileActionDeleted:
return "" // No preview URL for deleted files
default:
logging.FromContext(ctx).Error("unknown file action for preview URL", "action", f.Action)
return ""
}
}
// previewURL returns the URL to preview the file in Grafana
func (p *Previewer) previewURL(u *url.URL, repoName, ref, filePath, pullRequestURL string) string {
baseURL := *u
baseURL = *baseURL.JoinPath("/admin/provisioning", repoName, "dashboard/preview", filePath)
query := baseURL.Query()
if ref != "" {
query.Set("ref", ref)
}
if pullRequestURL != "" {
query.Set("pull_request_url", url.QueryEscape(pullRequestURL))
}
baseURL.RawQuery = query.Encode()
return baseURL.String()
}
// Preview creates a preview for a single file change
func (p *Previewer) Preview(
ctx context.Context,
f repository.VersionedFileChange,
namespace string,
repoName string,
base string,
ref string,
pullRequestURL string,
generatePreview bool,
) (*resourcePreview, error) {
baseURL, err := url.Parse(p.urlProvider(namespace))
if err != nil {
return nil, fmt.Errorf("error parsing base url: %w", err)
}
preview := resourcePreview{
Filename: path.Base(f.Path),
Path: f.Path,
Kind: "dashboard", // TODO: add more kinds
Action: string(f.Action),
OriginalURL: p.getOriginalURL(ctx, f, baseURL, repoName, base, pullRequestURL),
PreviewURL: p.getPreviewURL(ctx, f, baseURL, repoName, ref, pullRequestURL),
}
if !generatePreview && len(preview.PreviewURL) == 0 {
logger.Info("skipping dashboard preview generation", "path", f.Path)
return &preview, nil
}
screenshotURL, err := p.renderer.RenderDashboardPreview(ctx, namespace, repoName, f.Path, ref)
if err != nil {
return nil, fmt.Errorf("render dashboard preview: %w", err)
}
preview.PreviewScreenshotURL = screenshotURL
logger.Info("dashboard preview generated", "screenshotURL", screenshotURL)
return &preview, nil
}
pkg/registry/apis/provisioning/jobs/pullrequest/render.go
+93
View File
@@ -0,0 +1,93 @@
package pullrequest
import (
"context"
"fmt"
"mime"
"os"
"path/filepath"
"strings"
"time"
"github.com/grafana/grafana/pkg/apimachinery/identity"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/models"
"github.com/grafana/grafana/pkg/services/rendering"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
type screenshotRenderer struct {
render rendering.Service
blobstore resource.BlobStoreClient
urlProvider func(namespace string) string
isPublic bool
}
func NewScreenshotRenderer(render rendering.Service, blobstore resource.BlobStoreClient, isPublic bool, urlProvider func(namespace string) string) *screenshotRenderer {
return &screenshotRenderer{
render: render,
blobstore: blobstore,
urlProvider: urlProvider,
isPublic: isPublic,
}
}
func (r *screenshotRenderer) IsAvailable(ctx context.Context) bool {
return r.render != nil && r.render.IsAvailable(ctx) && r.blobstore != nil && r.isPublic
}
func (r *screenshotRenderer) RenderDashboardPreview(ctx context.Context, namespace, repoName, path, ref string) (string, error) {
url := fmt.Sprintf("admin/provisioning/%s/dashboard/preview/%s?kiosk&ref=%s", repoName, path, ref)
// TODO: why were we using a different context?
// renderContext := identity.WithRequester(context.Background(), r.id)
result, err := r.render.Render(ctx, rendering.RenderPNG, rendering.Opts{
CommonOpts: rendering.CommonOpts{
Path: url,
AuthOpts: rendering.AuthOpts{
OrgID: 1, // TODO!!!, use the worker identity
UserID: 1,
OrgRole: identity.RoleAdmin,
},
TimeoutOpts: rendering.TimeoutOpts{
Timeout: time.Second * 30,
},
},
Theme: models.ThemeDark, // from config?
Width: 1024,
Height: -1, // full page height
}, nil)
if err != nil {
return "", err
}
ext := filepath.Ext(result.FilePath)
body, err := os.ReadFile(result.FilePath)
if err != nil {
return "", err
}
rsp, err := r.blobstore.PutBlob(ctx, &resource.PutBlobRequest{
Resource: &resource.ResourceKey{
Namespace: namespace,
Group: provisioning.GROUP,
Resource: provisioning.RepositoryResourceInfo.GroupResource().Resource,
Name: repoName,
},
Method: resource.PutBlobRequest_GRPC,
ContentType: mime.TypeByExtension(ext), // image/png
Value: body,
})
if err != nil {
return "", err
}
if rsp.Url != "" {
return rsp.Url, nil
}
base := r.urlProvider(namespace)
if !strings.HasSuffix(base, "/") {
base += "/"
}
return fmt.Sprintf("%sapis/%s/namespaces/%s/repositories/%s/render/%s",
base, provisioning.APIVERSION, namespace, repoName, rsp.Uid), nil
}
pkg/registry/apis/provisioning/jobs/pullrequest/worker.go
+158
View File
@@ -0,0 +1,158 @@
package pullrequest
import (
"context"
"errors"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"github.com/grafana/grafana-app-sdk/logging"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
)
type PullRequestRepo interface {
Config() *provisioning.Repository
Read(ctx context.Context, path, ref string) (*repository.FileInfo, error)
CompareFiles(ctx context.Context, base, ref string) ([]repository.VersionedFileChange, error)
ClearAllPullRequestFileComments(ctx context.Context, pr int) error
CommentPullRequestFile(ctx context.Context, pr int, path string, ref string, comment string) error
CommentPullRequest(ctx context.Context, pr int, comment string) error
}
type PullRequestWorker struct {
parsers *resources.ParserFactory
previewer *Previewer
}
func NewPullRequestWorker(
parsers *resources.ParserFactory,
previewer *Previewer,
) (*PullRequestWorker, error) {
return &PullRequestWorker{
parsers: parsers,
previewer: previewer,
}, nil
}
func (c *PullRequestWorker) IsSupported(ctx context.Context, job provisioning.Job) bool {
return job.Spec.Action == provisioning.JobActionPullRequest
}
//nolint:gocyclo
func (c *PullRequestWorker) Process(ctx context.Context,
repo repository.Repository,
job provisioning.Job,
progress jobs.JobProgressRecorder,
) error {
cfg := repo.Config().Spec
options := job.Spec.PullRequest
if options == nil {
return apierrors.NewBadRequest("missing spec.pr")
}
prRepo, ok := repo.(PullRequestRepo)
if !ok {
return fmt.Errorf("repository is not a github repository")
}
reader, ok := repo.(repository.Reader)
if !ok {
return errors.New("pull request job submitted targeting repository that is not a Reader")
}
parser, err := c.parsers.GetParser(ctx, reader)
if err != nil {
return fmt.Errorf("failed to get parser for %s: %w", repo.Config().Name, err)
}
logger := logging.FromContext(ctx).With("pr", options.PR)
logger.Info("process pull request")
defer logger.Info("pull request processed")
progress.SetMessage(ctx, "listing pull request files")
base := cfg.GitHub.Branch
ref := options.Hash
files, err := prRepo.CompareFiles(ctx, base, ref)
if err != nil {
return fmt.Errorf("failed to list pull request files: %s", err.Error())
}
progress.SetMessage(ctx, "clearing pull request comments")
if err := prRepo.ClearAllPullRequestFileComments(ctx, options.PR); err != nil {
return fmt.Errorf("failed to clear pull request comments: %+v", err)
}
if len(files) == 0 {
progress.SetFinalMessage(ctx, "no files to process")
return nil
}
progress.SetMessage(ctx, "processing pull request files")
previews := make([]resourcePreview, 0, len(files))
for _, f := range files {
result := jobs.JobResourceResult{
Path: f.Path,
}
if err := resources.IsPathSupported(f.Path); err != nil {
result.Action = repository.FileActionIgnored
progress.Record(ctx, result)
continue
}
result.Action = f.Action
fileInfo, err := prRepo.Read(ctx, f.Path, ref)
if err != nil {
return fmt.Errorf("read file: %w", err)
}
parsed, err := parser.Parse(ctx, fileInfo, true)
if err != nil {
if errors.Is(err, resources.ErrUnableToReadResourceBytes) {
logger.Debug("file is not a resource", "path", f.Path)
result.Action = repository.FileActionIgnored
progress.Record(ctx, result)
} else {
result.Error = fmt.Errorf("failed to parse resource: %w", err)
progress.Record(ctx, result)
}
continue
}
result.Resource = parsed.GVR.Resource
result.Group = parsed.GVR.Group
result.Name = parsed.Obj.GetName()
preview, err := c.previewer.Preview(ctx, f, job.Namespace, repo.Config().Name, cfg.GitHub.Branch, ref, options.URL, cfg.GitHub.GenerateDashboardPreviews)
if err != nil {
result.Error = fmt.Errorf("create preview: %w", err)
progress.Record(ctx, result)
continue
}
previews = append(previews, *preview)
progress.Record(ctx, result)
}
if len(previews) == 0 {
progress.SetFinalMessage(ctx, "no previews to add")
return nil
}
progress.SetMessage(ctx, "generating previews comment")
comment, err := c.previewer.GenerateComment(previews)
if err != nil {
return fmt.Errorf("generate comment: %w", err)
}
if err := prRepo.CommentPullRequest(ctx, options.PR, comment); err != nil {
return fmt.Errorf("comment pull request: %w", err)
}
logger.Info("previews comment added", "number", len(previews))
return nil
}
pkg/registry/apis/provisioning/jobs/queue.go
+7 -19
View File
@@ -11,35 +11,23 @@ type RepoGetter interface {
GetRepository(ctx context.Context, name string) (repository.Repository, error)
}
// Basic job queue infrastructure
type JobQueue interface {
// Add a new Job to the Queue. The status must be empty
Add(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error)
// Get the next job we should process
Next(ctx context.Context) *provisioning.Job
// Update the status on a given job
// This is only valid if current job is not finished
Update(ctx context.Context, namespace string, name string, status provisioning.JobStatus) error
// Register a worker (inline for now)
Register(worker Worker)
}
type JobProgressRecorder interface {
Record(ctx context.Context, result JobResourceResult)
SetMessage(msg string)
GetMessage() string
ResetResults()
SetFinalMessage(ctx context.Context, msg string)
SetMessage(ctx context.Context, msg string)
SetRef(ref string)
GetRef() string
SetTotal(total int)
SetTotal(ctx context.Context, total int)
TooManyErrors() error
Complete(ctx context.Context, err error) provisioning.JobStatus
}
type Worker interface {
IsSupported(ctx context.Context, job provisioning.Job) bool
// Process the job. The job status should be updated as the job progresses.
//
// The job spec and metadata MUST not be modified in the storage layer while this is running. All updates go via the progress type.
Process(ctx context.Context, repo repository.Repository, job provisioning.Job, progress JobProgressRecorder) error
}
pkg/registry/apis/provisioning/jobs/store.go
-367
View File
@@ -1,367 +0,0 @@
package jobs
import (
"context"
"errors"
"fmt"
"net/http"
"strconv"
"sync"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
"k8s.io/apimachinery/pkg/apis/meta/internalversion"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/fields"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/watch"
"k8s.io/apiserver/pkg/endpoints/request"
"k8s.io/apiserver/pkg/registry/rest"
"k8s.io/apiserver/pkg/storage"
"github.com/grafana/grafana-app-sdk/logging"
"github.com/grafana/grafana/pkg/apimachinery/identity"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/util"
)
var (
_ JobQueue = (*jobStore)(nil)
_ rest.Scoper = (*jobStore)(nil)
_ rest.SingularNameProvider = (*jobStore)(nil)
_ rest.Getter = (*jobStore)(nil)
_ rest.Lister = (*jobStore)(nil)
_ rest.Storage = (*jobStore)(nil)
_ rest.Watcher = (*jobStore)(nil)
)
func NewJobStore(capacity int, getter RepoGetter) *jobStore {
return &jobStore{
workers: make([]Worker, 0),
getter: getter,
rv: 1,
capacity: capacity,
jobs: []provisioning.Job{},
watchSet: NewWatchSet(),
versioner: &storage.APIObjectVersioner{},
}
}
type jobStore struct {
getter RepoGetter
capacity int
workers []Worker
// All jobs
jobs []provisioning.Job
rv int64 // updates whenever changed
watchSet *WatchSet
versioner storage.Versioner
mutex sync.RWMutex
}
// Implementing Kube interfaces
func (s *jobStore) New() runtime.Object {
return provisioning.JobResourceInfo.NewFunc()
}
func (s *jobStore) Destroy() {}
func (s *jobStore) NamespaceScoped() bool {
return true // namespace == org
}
func (s *jobStore) GetSingularName() string {
return provisioning.JobResourceInfo.GetSingularName()
}
func (s *jobStore) NewList() runtime.Object {
return provisioning.JobResourceInfo.NewListFunc()
}
func (s *jobStore) ConvertToTable(ctx context.Context, object runtime.Object, tableOptions runtime.Object) (*metav1.Table, error) {
return provisioning.JobResourceInfo.TableConverter().ConvertToTable(ctx, object, tableOptions)
}
func (s *jobStore) List(ctx context.Context, options *internalversion.ListOptions) (runtime.Object, error) {
ns, ok := request.NamespaceFrom(ctx)
if !ok {
return nil, fmt.Errorf("missing namespace")
}
queue := &provisioning.JobList{
ListMeta: metav1.ListMeta{
ResourceVersion: strconv.FormatInt(s.rv, 10),
},
}
query := options.LabelSelector
s.mutex.RLock()
defer s.mutex.RUnlock()
for _, job := range s.jobs {
if job.Namespace != ns {
continue
}
// maybe filter
if query != nil && !query.Matches(labels.Set(job.Labels)) {
continue
}
copy := job.DeepCopy()
queue.Items = append(queue.Items, *copy)
}
return queue, nil
}
func (s *jobStore) Get(ctx context.Context, name string, options *metav1.GetOptions) (runtime.Object, error) {
s.mutex.RLock()
defer s.mutex.RUnlock()
ns, ok := request.NamespaceFrom(ctx)
if !ok {
return nil, fmt.Errorf("missing namespace")
}
for _, job := range s.jobs {
if job.Name == name && job.Namespace == ns {
return job.DeepCopy(), nil
}
}
return nil, apierrors.NewNotFound(provisioning.JobResourceInfo.GroupResource(), name)
}
func (s *jobStore) Watch(ctx context.Context, opts *internalversion.ListOptions) (watch.Interface, error) {
ns, ok := request.NamespaceFrom(ctx)
if !ok {
return nil, fmt.Errorf("missing namespace")
}
p := storage.SelectionPredicate{
Label: labels.Everything(), // TODO... limit
Field: fields.Everything(),
}
// Can watch by label selection
jw := s.watchSet.newWatch(ctx, 0, p, s.versioner, &ns)
jw.Start()
return jw, nil
}
// Implementing JobQueue
// Register a worker (inline for now)
func (s *jobStore) Register(worker Worker) {
s.workers = append(s.workers, worker)
}
func (s *jobStore) Add(ctx context.Context, job *provisioning.Job) (*provisioning.Job, error) {
if job.Namespace == "" {
return nil, apierrors.NewBadRequest("missing metadata.namespace")
}
if job.Name != "" {
return nil, apierrors.NewBadRequest("name will always be generated")
}
if job.Spec.Repository == "" {
return nil, apierrors.NewBadRequest("missing spec.repository")
}
if job.Spec.Action == "" {
return nil, apierrors.NewBadRequest("missing spec.action")
}
if job.Spec.Action == provisioning.JobActionExport && job.Spec.Push == nil {
return nil, apierrors.NewBadRequest("missing spec.push")
}
if job.Spec.Action == provisioning.JobActionSync && job.Spec.Pull == nil {
return nil, apierrors.NewBadRequest("missing spec.pull")
}
// Only for add
if job.Status.State != "" {
return nil, apierrors.NewBadRequest("must add jobs with empty status")
}
if job.Labels == nil {
job.Labels = make(map[string]string)
}
job.Labels["repository"] = job.Spec.Repository // for now, make sure we can search Multi-tenant
job.Name = fmt.Sprintf("%s:%s:%s", job.Spec.Repository, job.Spec.Action, util.GenerateShortUID())
s.mutex.Lock()
defer s.mutex.Unlock()
s.rv++
job.ResourceVersion = strconv.FormatInt(s.rv, 10)
job.Status.State = provisioning.JobStatePending
job.CreationTimestamp = metav1.NewTime(time.Now())
jobs := make([]provisioning.Job, 0, len(s.jobs)+2)
jobs = append(jobs, *job)
for i, j := range s.jobs {
if i >= s.capacity {
// Remove the old jobs
s.watchSet.notifyWatchers(watch.Event{
Object: j.DeepCopyObject(),
Type: watch.Deleted,
}, nil)
continue
}
jobs = append(jobs, j)
}
// Send add event
s.watchSet.notifyWatchers(watch.Event{
Object: job.DeepCopyObject(),
Type: watch.Added,
}, nil)
// For now, start a thread processing each job
go s.drainPending()
s.jobs = jobs // replace existing list
return job, nil
}
// Reads the queue until no jobs remain
func (s *jobStore) drainPending() {
logger := logging.DefaultLogger.With("logger", "job-store")
ctx := logging.Context(context.Background(), logger)
var err error
for {
time.Sleep(time.Microsecond * 200)
job := s.Next(ctx)
if job == nil {
return // done
}
logger := logger.With("job", job.GetName(), "namespace", job.GetNamespace())
ctx := logging.Context(ctx, logger)
var foundWorker bool
recorder := newJobProgressRecorder(func(ctx context.Context, j provisioning.JobStatus) error {
return s.Update(ctx, job.Namespace, job.Name, j)
})
for _, worker := range s.workers {
if !worker.IsSupported(ctx, *job) {
continue
}
// Already found a worker, no need to continue
foundWorker = true
err = s.processByWorker(ctx, worker, *job, recorder)
break
}
if !foundWorker {
err = errors.New("no registered worker supports this job")
}
status := recorder.Complete(ctx, err)
err = s.Update(ctx, job.Namespace, job.Name, status)
if err != nil {
logger.Error("error running job", "error", err)
}
logger.Debug("job has been fully completed")
}
}
func (s *jobStore) processByWorker(ctx context.Context, worker Worker, job provisioning.Job, recorder JobProgressRecorder) error {
ctx = request.WithNamespace(ctx, job.Namespace)
ctx, _, err := identity.WithProvisioningIdentitiy(ctx, job.Namespace)
if err != nil {
return fmt.Errorf("get worker identity: %w", err)
}
repoName := job.Spec.Repository
logger := logging.FromContext(ctx)
logger = logger.With("repository", repoName)
ctx = logging.Context(ctx, logger)
repo, err := s.getter.GetRepository(ctx, repoName)
if err != nil {
return fmt.Errorf("get repository: %w", err)
}
// TODO: does this really happen?
if repo == nil {
return errors.New("unknown repository")
}
return worker.Process(ctx, repo, job, recorder)
}
// Checkout the next "pending" job
func (s *jobStore) Next(ctx context.Context) *provisioning.Job {
s.mutex.Lock()
defer s.mutex.Unlock()
// The oldest jobs should be checked out first
for i := len(s.jobs) - 1; i >= 0; i-- {
if s.jobs[i].Status.State == provisioning.JobStatePending {
oldObj := s.jobs[i].DeepCopyObject()
s.rv++
s.jobs[i].ResourceVersion = strconv.FormatInt(s.rv, 10)
s.jobs[i].Status.State = provisioning.JobStateWorking
s.jobs[i].Status.Started = time.Now().UnixMilli()
job := s.jobs[i]
s.watchSet.notifyWatchers(watch.Event{
Object: job.DeepCopyObject(),
Type: watch.Modified,
}, oldObj)
return &job
}
}
return nil
}
func (s *jobStore) Update(ctx context.Context, namespace string, name string, status provisioning.JobStatus) error {
s.mutex.Lock()
defer s.mutex.Unlock()
s.rv++
if status.State == "" {
return apierrors.NewBadRequest("The state must be set")
}
if status.Progress > 100 || status.Progress < 0 {
return apierrors.NewBadRequest("progress must be between 0 and 100")
}
for idx, job := range s.jobs {
if job.Name == name && job.Namespace == namespace {
if job.Status.State.Finished() {
return &apierrors.StatusError{ErrStatus: metav1.Status{
Code: http.StatusPreconditionFailed,
Message: "The job is already finished and can not be updated",
}}
}
if status.State.Finished() {
status.Finished = time.Now().UnixMilli()
}
oldObj := job.DeepCopyObject()
job.ResourceVersion = strconv.FormatInt(s.rv, 10)
job.Status = status
s.jobs[idx] = job
s.watchSet.notifyWatchers(watch.Event{
Object: job.DeepCopyObject(),
Type: watch.Modified,
}, oldObj)
return nil
}
}
return apierrors.NewNotFound(provisioning.JobResourceInfo.GroupResource(), name)
}
pkg/registry/apis/provisioning/jobs/sync/changes.go
+88
View File
@@ -0,0 +1,88 @@
package sync
import (
"fmt"
"sort"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
type ResourceFileChange struct {
// Path to the file in a repository with a change
Path string
Action repository.FileAction
// The current value in the database -- only required for delete
Existing *provisioning.ResourceListItem
}
func Changes(source []repository.FileTreeEntry, target *provisioning.ResourceList) ([]ResourceFileChange, error) {
lookup := make(map[string]*provisioning.ResourceListItem, len(target.Items))
for _, item := range target.Items {
if item.Path == "" {
if item.Group != folders.GROUP {
return nil, fmt.Errorf("empty path on a non folder")
}
continue
}
lookup[item.Path] = &item
}
keep := safepath.NewTrie()
changes := make([]ResourceFileChange, 0, len(source))
for _, file := range source {
if !file.Blob {
continue // skip folder references?
}
check, ok := lookup[file.Path]
if ok {
if check.Hash != file.Hash {
changes = append(changes, ResourceFileChange{
Action: repository.FileActionUpdated,
Path: check.Path,
Existing: check,
})
}
if err := keep.Add(file.Path); err != nil {
return nil, fmt.Errorf("failed to add path to keep trie: %w", err)
}
delete(lookup, file.Path)
continue
}
// TODO: does this work with empty folders?
if resources.IsPathSupported(file.Path) == nil {
changes = append(changes, ResourceFileChange{
Action: repository.FileActionCreated, // or previously ignored/failed
Path: file.Path,
})
}
}
// Paths found in grafana, without a matching path in the repository
for _, v := range lookup {
if v.Resource == folders.RESOURCE && keep.Exists(v.Path) {
continue
}
changes = append(changes, ResourceFileChange{
Action: repository.FileActionDeleted,
Path: v.Path,
Existing: v,
})
}
// Deepest first (stable sort order)
sort.Slice(changes, func(i, j int) bool {
return safepath.Depth(changes[i].Path) > safepath.Depth(changes[j].Path)
})
return changes, nil
}
pkg/registry/apis/provisioning/jobs/sync/changes_test.go
+145
View File
@@ -0,0 +1,145 @@
package sync
import (
"encoding/json"
"testing"
"github.com/stretchr/testify/require"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
func TestChanges(t *testing.T) {
t.Run("start the same", func(t *testing.T) {
source, target := getBase(t)
changes, err := Changes(source, target)
require.NoError(t, err)
require.Empty(t, changes)
})
t.Run("create a source file", func(t *testing.T) {
source, target := getBase(t)
source = append(source, repository.FileTreeEntry{
Path: "muta.json", Hash: "xyz", Blob: true,
})
changes, err := Changes(source, target)
require.NoError(t, err)
require.Len(t, changes, 1)
require.Equal(t, ResourceFileChange{
Action: repository.FileActionCreated,
Path: "muta.json",
}, changes[0])
})
t.Run("delete a source file", func(t *testing.T) {
source, target := getBase(t)
source = []repository.FileTreeEntry{source[0]}
changes, err := Changes(source, target)
require.NoError(t, err)
require.Len(t, changes, 1)
require.Equal(t, ResourceFileChange{
Action: repository.FileActionDeleted,
Path: "adsl62h.yaml",
Existing: &provisioning.ResourceListItem{
Path: "adsl62h.yaml",
Group: "dashboard.grafana.app",
Resource: "dashboards",
Name: "adsl62h-hrw-f-fvlt2dghp-gufrc4lisksgmq-c",
Hash: "ce5d497c4deadde6831162ce8509e2b2b1776237",
},
}, changes[0])
})
t.Run("folder deletion order", func(t *testing.T) {
source := []repository.FileTreeEntry{
{Path: "x/y/z/ignored.md"}, // ignored
{Path: "aaa/bbb.yaml", Hash: "xyz", Blob: true},
}
target := &provisioning.ResourceList{
Items: []provisioning.ResourceListItem{
{Path: "a.json"},
{Path: "x/y/file.json"},
{Path: "aaa/", Resource: "folders"}, // Folder... no action required
{Path: "aaa/bbb.yaml", Hash: "xyz"},
{Path: "zzz/longest/path/here.json"},
{Path: "short/file.yml"},
},
}
changes, err := Changes(source, target)
require.NoError(t, err)
order := make([]string, len(changes))
for i := range changes {
order[i] = changes[i].Path
}
require.Equal(t, []string{
"zzz/longest/path/here.json", // not sorted yet
"x/y/file.json",
"short/file.yml",
"a.json",
}, order)
})
t.Run("modify a file", func(t *testing.T) {
source, target := getBase(t)
source[1].Hash = "different"
changes, err := Changes(source, target)
require.NoError(t, err)
require.Len(t, changes, 1)
require.Equal(t, ResourceFileChange{
Action: repository.FileActionUpdated,
Path: "adsl62h.yaml",
Existing: &provisioning.ResourceListItem{
Path: "adsl62h.yaml",
Group: "dashboard.grafana.app",
Resource: "dashboards",
Name: "adsl62h-hrw-f-fvlt2dghp-gufrc4lisksgmq-c",
Hash: "ce5d497c4deadde6831162ce8509e2b2b1776237",
},
}, changes[0])
})
}
func getBase(t *testing.T) (source []repository.FileTreeEntry, target *provisioning.ResourceList) {
target = &provisioning.ResourceList{}
err := json.Unmarshal([]byte(`{
"kind": "ResourceList",
"apiVersion": "provisioning.grafana.app/v0alpha1",
"metadata": {},
"items": [
{
"path": "",
"group": "folder.grafana.app",
"resource": "folders",
"name": "simplelocal-3794ab9",
"hash": ""
},
{
"path": "ad4lwp2.yaml",
"group": "dashboard.grafana.app",
"resource": "dashboards",
"name": "ad4lwp2-xofjsuo-mr5blr1zwimlfi0ds0pyrrpd",
"hash": "ca83d64b9c4a23fed975aacdf47e7de8878b4ae0"
},
{
"path": "adsl62h.yaml",
"group": "dashboard.grafana.app",
"resource": "dashboards",
"name": "adsl62h-hrw-f-fvlt2dghp-gufrc4lisksgmq-c",
"hash": "ce5d497c4deadde6831162ce8509e2b2b1776237"
}
]
}`), target)
require.NoError(t, err)
source = []repository.FileTreeEntry{
{Path: "ad4lwp2.yaml", Hash: "ca83d64b9c4a23fed975aacdf47e7de8878b4ae0", Blob: true},
{Path: "adsl62h.yaml", Hash: "ce5d497c4deadde6831162ce8509e2b2b1776237", Blob: true},
}
return // named values!
}
pkg/registry/apis/provisioning/jobs/sync/worker.go
+476
View File
@@ -0,0 +1,476 @@
package sync
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"sort"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/types"
"k8s.io/client-go/dynamic"
"github.com/grafana/grafana-app-sdk/logging"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
client "github.com/grafana/grafana/pkg/generated/clientset/versioned/typed/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
)
// SyncWorker synchronizes the external repo with grafana database
// this function updates the status for both the job and the referenced repository
type SyncWorker struct {
// Used to update the repository status with sync info
client client.ProvisioningV0alpha1Interface
// Lists the values saved in grafana database
lister resources.ResourceLister
// Parses fields saved in remore repository
parsers *resources.ParserFactory
// Check if the system is using unified storage
storageStatus dualwrite.Service
}
func NewSyncWorker(
client client.ProvisioningV0alpha1Interface,
parsers *resources.ParserFactory,
lister resources.ResourceLister,
storageStatus dualwrite.Service,
) *SyncWorker {
return &SyncWorker{
client: client,
parsers: parsers,
lister: lister,
storageStatus: storageStatus,
}
}
func (r *SyncWorker) IsSupported(ctx context.Context, job provisioning.Job) bool {
return job.Spec.Action == provisioning.JobActionSync
}
func (r *SyncWorker) Process(ctx context.Context, repo repository.Repository, job provisioning.Job, progress jobs.JobProgressRecorder) error {
cfg := repo.Config()
logger := logging.FromContext(ctx).With("job", job.GetName(), "namespace", job.GetNamespace())
// Check if we are onboarding from legacy storage
if dualwrite.IsReadingLegacyDashboardsAndFolders(ctx, r.storageStatus) {
return fmt.Errorf("sync not supported until storage has migrated")
}
rw, ok := repo.(repository.Reader)
if !ok {
return fmt.Errorf("sync job submitted for repository that does not support read-write -- this is a bug")
}
syncStatus := job.Status.ToSyncStatus(job.Name)
// Preserve last ref as we use replace operation
syncStatus.LastRef = repo.Config().Status.Sync.LastRef
// Update sync status at start using JSON patch
patchOperations := []map[string]interface{}{
{
"op": "replace",
"path": "/status/sync",
"value": syncStatus,
},
}
progress.SetMessage(ctx, "update sync status at start")
if err := r.patchStatus(ctx, cfg, patchOperations); err != nil {
return fmt.Errorf("update repo with job status at start: %w", err)
}
progress.SetMessage(ctx, "execute sync job")
syncJob, err := r.createJob(ctx, rw, progress)
if err != nil {
return fmt.Errorf("failed to create sync job: %w", err)
}
syncError := syncJob.run(ctx, *job.Spec.Pull)
jobStatus := progress.Complete(ctx, syncError)
syncStatus = jobStatus.ToSyncStatus(job.Name)
// Create sync status and set hash if successful
if syncStatus.State == provisioning.JobStateSuccess {
syncStatus.LastRef = progress.GetRef()
}
// Update final status using JSON patch
progress.SetMessage(ctx, "update status and stats")
patchOperations = []map[string]interface{}{
{
"op": "replace",
"path": "/status/sync",
"value": syncStatus,
},
}
// Only add stats patch if stats are not nil
if stats, err := r.lister.Stats(ctx, cfg.Namespace, cfg.Name); err != nil {
logger.Error("unable to read stats", "error", err)
} else if stats != nil && len(stats.Managed) == 1 {
patchOperations = append(patchOperations, map[string]interface{}{
"op": "replace",
"path": "/status/stats",
"value": stats.Managed[0].Stats,
})
}
// Only patch the specific fields we want to update, not the entire status
if err := r.patchStatus(ctx, cfg, patchOperations); err != nil {
return fmt.Errorf("update repo with job final status: %w", err)
}
return syncError
}
// start a job and run it
func (r *SyncWorker) createJob(ctx context.Context, repo repository.Reader, progress jobs.JobProgressRecorder) (*syncJob, error) {
cfg := repo.Config()
parser, err := r.parsers.GetParser(ctx, repo)
if err != nil {
return nil, fmt.Errorf("failed to get parser for %s: %w", cfg.Name, err)
}
folderClient, err := parser.Clients().Folder()
if err != nil {
return nil, fmt.Errorf("unable to get folder client: %w", err)
}
dashboardClient, err := parser.Clients().Dashboard()
if err != nil {
return nil, fmt.Errorf("unable to get dashboard client: %w", err)
}
job := &syncJob{
repository: repo,
progress: progress,
parser: parser,
lister: r.lister,
folders: resources.NewFolderManager(repo, folderClient),
dashboards: dashboardClient,
resourcesLookup: map[resourceID]string{},
}
return job, nil
}
func (r *SyncWorker) patchStatus(ctx context.Context, repo *provisioning.Repository, patchOperations []map[string]interface{}) error {
patch, err := json.Marshal(patchOperations)
if err != nil {
return fmt.Errorf("unable to marshal patch data: %w", err)
}
_, err = r.client.Repositories(repo.Namespace).
Patch(ctx, repo.Name, types.JSONPatchType, patch, metav1.PatchOptions{}, "status")
if err != nil {
return fmt.Errorf("unable to update repo with job status: %w", err)
}
return nil
}
type resourceID struct {
Name string
Resource string
Group string
}
// created once for each sync execution
type syncJob struct {
repository repository.Reader
progress jobs.JobProgressRecorder
parser *resources.Parser
lister resources.ResourceLister
folders *resources.FolderManager
dashboards dynamic.ResourceInterface
folderLookup *resources.FolderTree
resourcesLookup map[resourceID]string // the path with this k8s name
}
func (r *syncJob) run(ctx context.Context, options provisioning.SyncJobOptions) error {
// Ensure the configured folder exists and is managed by the repository
cfg := r.repository.Config()
rootFolder := resources.RootFolder(cfg)
if rootFolder != "" {
if err := r.folders.EnsureFolderExists(ctx, resources.Folder{
ID: rootFolder, // will not change if exists
Title: cfg.Spec.Title,
Path: "", // at the root of the repository
}, ""); err != nil {
return fmt.Errorf("unable to create root folder: %w", err)
}
}
var err error
var currentRef string
versionedRepo, _ := r.repository.(repository.Versioned)
if versionedRepo != nil {
currentRef, err = versionedRepo.LatestRef(ctx)
if err != nil {
return fmt.Errorf("getting latest ref: %w", err)
}
r.progress.SetRef(currentRef)
if cfg.Status.Sync.LastRef != "" && options.Incremental {
if currentRef == cfg.Status.Sync.LastRef {
r.progress.SetFinalMessage(ctx, "same commit as last sync")
return nil
}
return r.applyVersionedChanges(ctx, versionedRepo, cfg.Status.Sync.LastRef, currentRef)
}
}
// Read the complete change set
target, err := r.lister.List(ctx, cfg.Namespace, cfg.Name)
if err != nil {
return fmt.Errorf("error listing current: %w", err)
}
source, err := r.repository.ReadTree(ctx, currentRef)
if err != nil {
return fmt.Errorf("error reading tree: %w", err)
}
changes, err := Changes(source, target)
if err != nil {
return fmt.Errorf("error calculating changes: %w", err)
}
if len(changes) == 0 {
r.progress.SetFinalMessage(ctx, "no changes to sync")
return nil
}
// Load any existing folder information
r.folderLookup = resources.NewFolderTreeFromResourceList(target)
// Now apply the changes
return r.applyChanges(ctx, changes)
}
func (r *syncJob) applyChanges(ctx context.Context, changes []ResourceFileChange) error {
if len(r.resourcesLookup) > 0 {
return fmt.Errorf("this should be empty")
}
// Do the deepest paths first (important for delete)
sort.Slice(changes, func(i, j int) bool {
return safepath.Depth(changes[i].Path) > safepath.Depth(changes[j].Path)
})
r.progress.SetTotal(ctx, len(changes))
r.progress.SetMessage(ctx, "replicating changes")
// Create folder structure first
for _, change := range changes {
if err := r.progress.TooManyErrors(); err != nil {
return err
}
if change.Action == repository.FileActionDeleted {
result := jobs.JobResourceResult{
Name: change.Existing.Name,
Resource: change.Existing.Resource,
Group: change.Existing.Group,
Path: change.Path,
Action: change.Action,
}
if change.Existing == nil || change.Existing.Name == "" {
result.Error = errors.New("missing existing reference")
r.progress.Record(ctx, result)
continue
}
client, err := r.client(change.Existing.Resource)
if err != nil {
result.Error = fmt.Errorf("unable to get client for deleted object: %w", err)
r.progress.Record(ctx, result)
continue
}
result.Error = client.Delete(ctx, change.Existing.Name, metav1.DeleteOptions{})
r.progress.Record(ctx, result)
continue
}
// Write the resource file
r.progress.Record(ctx, r.writeResourceFromFile(ctx, change.Path, "", change.Action))
}
r.progress.SetMessage(ctx, "changes replicated")
return nil
}
// Convert git changes into resource file changes
func (r *syncJob) applyVersionedChanges(ctx context.Context, repo repository.Versioned, previousRef, currentRef string) error {
diff, err := repo.CompareFiles(ctx, previousRef, currentRef)
if err != nil {
return fmt.Errorf("compare files error: %w", err)
}
if len(diff) < 1 {
r.progress.SetFinalMessage(ctx, "no changes detected between commits")
return nil
}
r.progress.SetTotal(ctx, len(diff))
r.progress.SetMessage(ctx, "replicating versioned changes")
for _, change := range diff {
if err := r.progress.TooManyErrors(); err != nil {
return err
}
if err := resources.IsPathSupported(change.Path); err != nil {
r.progress.Record(ctx, jobs.JobResourceResult{
Path: change.Path,
Action: repository.FileActionIgnored,
})
continue
}
switch change.Action {
case repository.FileActionCreated, repository.FileActionUpdated:
r.progress.Record(ctx, r.writeResourceFromFile(ctx, change.Path, change.Ref, change.Action))
case repository.FileActionDeleted:
r.progress.Record(ctx, r.deleteObject(ctx, change.Path, change.PreviousRef))
case repository.FileActionRenamed:
// 1. Delete
result := r.deleteObject(ctx, change.Path, change.PreviousRef)
if result.Error != nil {
r.progress.Record(ctx, result)
continue
}
// 2. Create
r.progress.Record(ctx, r.writeResourceFromFile(ctx, change.Path, change.Ref, repository.FileActionCreated))
case repository.FileActionIgnored:
r.progress.Record(ctx, jobs.JobResourceResult{
Path: change.Path,
Action: repository.FileActionIgnored,
})
}
}
r.progress.SetMessage(ctx, "versioned changes replicated")
return nil
}
func (r *syncJob) deleteObject(ctx context.Context, path string, ref string) jobs.JobResourceResult {
info, err := r.repository.Read(ctx, path, ref)
result := jobs.JobResourceResult{
Path: path,
Action: repository.FileActionDeleted,
}
if err != nil {
result.Error = fmt.Errorf("failed to read file: %w", err)
return result
}
obj, gvk, _ := resources.DecodeYAMLObject(bytes.NewBuffer(info.Data))
if obj == nil {
result.Error = errors.New("no object found")
return result
}
objName := obj.GetName()
if objName == "" {
// Find the referenced file
objName, _ = resources.NamesFromHashedRepoPath(r.repository.Config().Name, path)
}
result.Name = objName
result.Resource = gvk.Kind
result.Group = gvk.Group
client, err := r.client(gvk.Kind)
if err != nil {
result.Error = fmt.Errorf("unable to get client for deleted object: %w", err)
return result
}
err = client.Delete(ctx, objName, metav1.DeleteOptions{})
if err != nil {
result.Error = fmt.Errorf("failed to delete: %w", err)
return result
}
return result
}
func (r *syncJob) writeResourceFromFile(ctx context.Context, path string, ref string, action repository.FileAction) jobs.JobResourceResult {
result := jobs.JobResourceResult{
Path: path,
Action: action,
}
// Read the referenced file
fileInfo, err := r.repository.Read(ctx, path, ref)
if err != nil {
result.Error = fmt.Errorf("failed to read file: %w", err)
return result
}
parsed, err := r.parser.Parse(ctx, fileInfo, false) // no validation
if err != nil {
result.Error = fmt.Errorf("failed to parse file: %w", err)
return result
}
// Check if the resource already exists
id := resourceID{
Name: parsed.Obj.GetName(),
Resource: parsed.GVR.Resource,
Group: parsed.GVK.Group,
}
existing, found := r.resourcesLookup[id]
if found {
result.Error = fmt.Errorf("duplicate resource name: %s, %s and %s", parsed.Obj.GetName(), path, existing)
return result
}
r.resourcesLookup[id] = path
// Make sure the parent folders exist
folder, err := r.folders.EnsureFolderPathExist(ctx, path)
if err != nil {
result.Error = fmt.Errorf("failed to ensure folder path exists: %w", err)
return result
}
parsed.Meta.SetFolder(folder)
parsed.Meta.SetUID("") // clear identifiers
parsed.Meta.SetResourceVersion("") // clear identifiers
result.Name = parsed.Obj.GetName()
result.Resource = parsed.GVR.Resource
result.Group = parsed.GVK.Group
// Update will also create (for resources we care about)
_, err = parsed.Client.Update(ctx, parsed.Obj, metav1.UpdateOptions{})
result.Error = err
return result
}
func (r *syncJob) client(kind string) (dynamic.ResourceInterface, error) {
switch kind {
case dashboard.GROUP, dashboard.DASHBOARD_RESOURCE, "Dashboard":
return r.dashboards, nil
case folders.GROUP, folders.RESOURCE, "Folder":
return r.folders.Client(), nil
}
return nil, fmt.Errorf("unsupported resource: %s", kind)
}
pkg/registry/apis/provisioning/jobs/watchset.go
-379
View File
@@ -1,379 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Provenance-includes-location: https://github.com/tilt-dev/tilt-apiserver/blob/main/pkg/storage/filepath/watchset.go
// Provenance-includes-license: Apache-2.0
// Provenance-includes-copyright: The Kubernetes Authors.
// See also
// https://github.com/grafana/grafana/blob/v11.1.9/pkg/apiserver/storage/file/watchset.go
package jobs
import (
"context"
"fmt"
"sync"
"sync/atomic"
"k8s.io/apimachinery/pkg/api/meta"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/watch"
"k8s.io/apiserver/pkg/storage"
"k8s.io/klog/v2"
)
const (
UpdateChannelSize = 25
InitialWatchNodesSize = 20
InitialBufferedEventsSize = 25
)
type eventWrapper struct {
ev watch.Event
// optional: oldObject is only set for modifications for determining their type as necessary (when using predicate filtering)
oldObject runtime.Object
}
type watchNode struct {
ctx context.Context
s *WatchSet
id uint64
updateCh chan eventWrapper
outCh chan watch.Event
requestedRV uint64
// the watch may or may not be namespaced for a namespaced resource. This is always nil for cluster-scoped kinds
watchNamespace *string
predicate storage.SelectionPredicate
versioner storage.Versioner
}
// Keeps track of which watches need to be notified
type WatchSet struct {
mu sync.RWMutex
// mu protects both nodes and counter
nodes map[uint64]*watchNode
counter atomic.Uint64
buffered []eventWrapper
bufferedMutex sync.RWMutex
}
func NewWatchSet() *WatchSet {
return &WatchSet{
buffered: make([]eventWrapper, 0, InitialBufferedEventsSize),
nodes: make(map[uint64]*watchNode, InitialWatchNodesSize),
}
}
// Creates a new watch with a unique id, but
// does not start sending events to it until start() is called.
func (s *WatchSet) newWatch(ctx context.Context, requestedRV uint64, p storage.SelectionPredicate, versioner storage.Versioner, namespace *string) *watchNode {
s.counter.Add(1)
node := &watchNode{
ctx: ctx,
requestedRV: requestedRV,
id: s.counter.Load(),
s: s,
// updateCh size needs to be > 1 to allow slower clients to not block passing new events
updateCh: make(chan eventWrapper, UpdateChannelSize),
// outCh size needs to be > 1 for single process use-cases such as tests where watch and event seeding from CUD
// events is happening on the same thread
outCh: make(chan watch.Event, UpdateChannelSize),
predicate: p,
watchNamespace: namespace,
versioner: versioner,
}
return node
}
func (s *WatchSet) CleanupWatchers() {
s.mu.Lock()
defer s.mu.Unlock()
for _, w := range s.nodes {
w.stop()
}
}
// oldObject is only passed in the event of a modification
// in case a predicate filtered watch is impacted as a result of modification
// NOTE: this function gives one the misperception that a newly added node will never
// get a double event, one from buffered and one from the update channel
// That perception is not true. Even though this function maintains the lock throughout the function body
// it is not true of the Start function. So basically, the Start function running after this function
// fully stands the chance of another future notifyWatchers double sending it the event through the two means mentioned
func (s *WatchSet) notifyWatchers(ev watch.Event, oldObject runtime.Object) {
s.mu.RLock()
defer s.mu.RUnlock()
updateEv := eventWrapper{
ev: ev,
}
if oldObject != nil {
updateEv.oldObject = oldObject
}
// Events are always buffered.
// this is because of an inadvertent delay which is built into the watch process
// Watch() from storage returns Watch.Interface with a async start func.
// The only way to guarantee that we can interpret the passed RV correctly is to play it against missed events
// (notice the loop below over s.nodes isn't exactly going to work on a new node
// unless start is called on it)
s.bufferedMutex.Lock()
s.buffered = append(s.buffered, updateEv)
s.bufferedMutex.Unlock()
for _, w := range s.nodes {
w.updateCh <- updateEv
}
}
// isValid is not necessary to be called on oldObject in UpdateEvents - assuming the Watch pushes correctly setup eventWrapper our way
// first bool is whether the event is valid for current watcher
// second bool is whether checking the old value against the predicate may be valuable to the caller
// second bool may be a helpful aid to establish context around MODIFIED events
// (note that this second bool is only marked true if we pass other checks first, namely RV and namespace)
func (w *watchNode) isValid(e eventWrapper) (bool, bool, error) {
obj, err := meta.Accessor(e.ev.Object)
if err != nil {
klog.Error("Could not get accessor to object in event")
return false, false, nil
}
eventRV, err := w.getResourceVersionAsInt(e.ev.Object)
if err != nil {
return false, false, err
}
if eventRV < w.requestedRV {
return false, false, nil
}
if w.watchNamespace != nil && *w.watchNamespace != obj.GetNamespace() {
return false, false, err
}
valid, err := w.predicate.Matches(e.ev.Object)
if err != nil {
return false, false, err
}
return valid, e.ev.Type == watch.Modified, nil
}
// Only call this method if current object matches the predicate
func (w *watchNode) handleAddedForFilteredList(e eventWrapper) (*watch.Event, error) {
if e.oldObject == nil {
return nil, fmt.Errorf("oldObject should be set for modified events")
}
ok, err := w.predicate.Matches(e.oldObject)
if err != nil {
return nil, err
}
if !ok {
e.ev.Type = watch.Added
return &e.ev, nil
}
return nil, nil
}
func (w *watchNode) handleDeletedForFilteredList(e eventWrapper) (*watch.Event, error) {
if e.oldObject == nil {
return nil, fmt.Errorf("oldObject should be set for modified events")
}
ok, err := w.predicate.Matches(e.oldObject)
if err != nil {
return nil, err
}
if !ok {
return nil, nil
}
// isn't a match but used to be
e.ev.Type = watch.Deleted
oldObjectAccessor, err := meta.Accessor(e.oldObject)
if err != nil {
klog.Errorf("Could not get accessor to correct the old RV of filtered out object")
return nil, err
}
currentRV, err := getResourceVersion(e.ev.Object)
if err != nil {
klog.Errorf("Could not get accessor to object in event")
return nil, err
}
oldObjectAccessor.SetResourceVersion(currentRV)
e.ev.Object = e.oldObject
return &e.ev, nil
}
func (w *watchNode) processEvent(e eventWrapper, isInitEvent bool) error {
if isInitEvent {
// Init events have already been vetted against the predicate and other RV behavior
// Let them pass through
w.outCh <- e.ev
return nil
}
valid, runDeleteFromFilteredListHandler, err := w.isValid(e)
if err != nil {
klog.Errorf("Could not determine validity of the event: %v", err)
return err
}
if valid {
if e.ev.Type == watch.Modified {
ev, err := w.handleAddedForFilteredList(e)
if err != nil {
return err
}
if ev != nil {
w.outCh <- *ev
} else {
// forward the original event if add handling didn't signal any impact
w.outCh <- e.ev
}
} else {
w.outCh <- e.ev
}
return nil
}
if runDeleteFromFilteredListHandler {
if e.ev.Type == watch.Modified {
ev, err := w.handleDeletedForFilteredList(e)
if err != nil {
return err
}
if ev != nil {
w.outCh <- *ev
}
} // explicitly doesn't have an event forward for the else case here
return nil
}
return nil
}
// Start sending events to this watch.
func (w *watchNode) Start(initEvents ...watch.Event) {
w.s.mu.Lock()
w.s.nodes[w.id] = w
w.s.mu.Unlock()
go func() {
maxRV := uint64(0)
for _, ev := range initEvents {
currentRV, err := w.getResourceVersionAsInt(ev.Object)
if err != nil {
klog.Errorf("Could not determine init event RV for deduplication of buffered events: %v", err)
continue
}
if maxRV < currentRV {
maxRV = currentRV
}
if err := w.processEvent(eventWrapper{ev: ev}, true); err != nil {
klog.Errorf("Could not process event: %v", err)
}
}
// If we had no init events, simply rely on the passed RV
if maxRV == 0 {
maxRV = w.requestedRV
}
w.s.bufferedMutex.RLock()
for _, e := range w.s.buffered {
eventRV, err := w.getResourceVersionAsInt(e.ev.Object)
if err != nil {
klog.Errorf("Could not determine RV for deduplication of buffered events: %v", err)
continue
}
if maxRV >= eventRV {
continue
} else {
maxRV = eventRV
}
if err := w.processEvent(e, false); err != nil {
klog.Errorf("Could not process event: %v", err)
}
}
w.s.bufferedMutex.RUnlock()
for {
select {
case e, ok := <-w.updateCh:
if !ok {
close(w.outCh)
return
}
eventRV, err := w.getResourceVersionAsInt(e.ev.Object)
if err != nil {
klog.Errorf("Could not determine RV for deduplication of channel events: %v", err)
continue
}
if maxRV >= eventRV {
continue
} else {
maxRV = eventRV
}
if err := w.processEvent(e, false); err != nil {
klog.Errorf("Could not process event: %v", err)
}
case <-w.ctx.Done():
close(w.outCh)
return
}
}
}()
}
func (w *watchNode) Stop() {
w.s.mu.Lock()
defer w.s.mu.Unlock()
w.stop()
}
// Unprotected func: ensure mutex on the parent watch set is locked before calling
func (w *watchNode) stop() {
if _, ok := w.s.nodes[w.id]; ok {
delete(w.s.nodes, w.id)
close(w.updateCh)
}
}
func (w *watchNode) ResultChan() <-chan watch.Event {
return w.outCh
}
func getResourceVersion(obj runtime.Object) (string, error) {
accessor, err := meta.Accessor(obj)
if err != nil {
klog.Error("Could not get accessor to object in event")
return "", err
}
return accessor.GetResourceVersion(), nil
}
func (w *watchNode) getResourceVersionAsInt(obj runtime.Object) (uint64, error) {
accessor, err := meta.Accessor(obj)
if err != nil {
klog.Error("Could not get accessor to object in event")
return 0, err
}
return w.versioner.ParseResourceVersion(accessor.GetResourceVersion())
}
pkg/registry/apis/provisioning/list.go
+66
View File
@@ -0,0 +1,66 @@
package provisioning
import (
"context"
"fmt"
"net/http"
"time"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/endpoints/request"
"k8s.io/apiserver/pkg/registry/rest"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/resources"
)
// TODO: Rename to resources as it's not clear that we are returning here is repository resources
type listConnector struct {
getter RepoGetter
lister resources.ResourceLister
}
func (*listConnector) New() runtime.Object {
return &provisioning.ResourceList{}
}
func (*listConnector) Destroy() {}
func (*listConnector) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (*listConnector) ProducesObject(verb string) any {
return &provisioning.ResourceList{}
}
func (*listConnector) ConnectMethods() []string {
return []string{http.MethodGet}
}
func (*listConnector) NewConnectOptions() (runtime.Object, bool, string) {
return nil, false, ""
}
func (s *listConnector) Connect(ctx context.Context, name string, opts runtime.Object, responder rest.Responder) (http.Handler, error) {
ns, ok := request.NamespaceFrom(ctx)
if !ok {
return nil, fmt.Errorf("missing namespace")
}
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
// TODO: Add pagination to resource lister
rsp, err := s.lister.List(ctx, ns, name)
if err != nil {
responder.Error(err)
} else {
responder.Object(200, rsp)
}
}), 30*time.Second), nil
}
var (
_ rest.Storage = (*listConnector)(nil)
_ rest.Connecter = (*listConnector)(nil)
_ rest.StorageMetadata = (*listConnector)(nil)
)
pkg/registry/apis/provisioning/migrate.go
+89
View File
@@ -0,0 +1,89 @@
package provisioning
import (
"context"
"net/http"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/registry/rest"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/jobs"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
)
// TODO: should we have merge migrate and sync connectors and have a single repository job connector?
type migrateConnector struct {
dual dualwrite.Service
repoGetter RepoGetter
jobs jobs.Queue
}
func (*migrateConnector) New() runtime.Object {
return &provisioning.Job{}
}
func (*migrateConnector) Destroy() {}
func (*migrateConnector) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (c *migrateConnector) ProducesObject(verb string) any {
return c.New()
}
func (*migrateConnector) ConnectMethods() []string {
return []string{http.MethodPost}
}
func (*migrateConnector) NewConnectOptions() (runtime.Object, bool, string) {
return nil, false, ""
}
func (c *migrateConnector) Connect(
ctx context.Context,
name string,
opts runtime.Object,
responder rest.Responder,
) (http.Handler, error) {
repo, err := c.repoGetter.GetHealthyRepository(ctx, name)
if err != nil {
return nil, err
}
cfg := repo.Config()
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
var options provisioning.MigrateJobOptions
if err := unmarshalJSON(r, defaultMaxBodySize, &options); err != nil {
responder.Error(apierrors.NewBadRequest("error decoding MigrateJobOptions from request"))
return
}
job, err := c.jobs.Insert(ctx, &provisioning.Job{
ObjectMeta: v1.ObjectMeta{
Namespace: cfg.Namespace,
},
Spec: provisioning.JobSpec{
Action: provisioning.JobActionMigrate,
Repository: cfg.Name,
Migrate: &options,
},
})
if err != nil {
responder.Error(err)
return
}
responder.Object(http.StatusAccepted, job)
}), 30*time.Second), nil
}
var (
_ rest.Connecter = (*migrateConnector)(nil)
_ rest.Storage = (*migrateConnector)(nil)
_ rest.StorageMetadata = (*migrateConnector)(nil)
)
pkg/registry/apis/provisioning/register.go
+968 -38
View File
File diff suppressed because it is too large Load Diff
pkg/registry/apis/provisioning/render.go
+131
View File
@@ -0,0 +1,131 @@
package provisioning
import (
"context"
"fmt"
"net/http"
"strings"
"time"
apierrors "k8s.io/apimachinery/pkg/api/errors"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apiserver/pkg/endpoints/request"
"k8s.io/apiserver/pkg/registry/rest"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/cmd/grafana-cli/logger"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
type renderConnector struct {
blob resource.BlobStoreClient
}
func (*renderConnector) New() runtime.Object {
return &provisioning.Repository{}
}
func (*renderConnector) Destroy() {}
func (*renderConnector) ProducesMIMETypes(verb string) []string {
return []string{"application/json"}
}
func (c *renderConnector) ProducesObject(verb string) any {
return c.New()
}
func (*renderConnector) ConnectMethods() []string {
return []string{http.MethodGet}
}
func (*renderConnector) NewConnectOptions() (runtime.Object, bool, string) {
return nil, true, ""
}
func (c *renderConnector) Connect(
ctx context.Context,
name string,
opts runtime.Object,
responder rest.Responder,
) (http.Handler, error) {
namespace := request.NamespaceValue(ctx)
return withTimeout(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
prefix := fmt.Sprintf("/%s/render", name)
idx := strings.Index(r.URL.Path, prefix)
if idx == -1 {
logger.Debug("failed to find a file path in the URL")
responder.Error(apierrors.NewBadRequest("invalid request path"))
return
}
blobID := strings.TrimPrefix(r.URL.Path[idx+len(prefix):], "/")
if len(blobID) == 0 {
responder.Error(apierrors.NewNotFound(provisioning.RepositoryResourceInfo.GroupResource(), "render"))
return
}
if !validBlobID(blobID) {
responder.Error(apierrors.NewBadRequest(fmt.Sprintf("invalid blob id: %s", blobID)))
return
}
rsp, err := c.blob.GetBlob(ctx, &resource.GetBlobRequest{
Resource: &resource.ResourceKey{
Namespace: namespace,
Group: provisioning.GROUP,
Resource: provisioning.RepositoryResourceInfo.GroupResource().Resource,
Name: name,
},
MustProxyBytes: true,
Uid: blobID,
})
if err != nil {
responder.Error(err)
return
}
if rsp.Error != nil {
responder.Error(resource.GetError(rsp.Error))
return
}
if len(rsp.Value) > 0 {
if rsp.ContentType != "" {
w.Header().Add("Content-Type", rsp.ContentType)
}
_, err = w.Write(rsp.Value)
if err != nil {
responder.Error(err)
return
}
} else {
responder.Error(&apierrors.StatusError{
ErrStatus: v1.Status{
Code: http.StatusNoContent,
Message: "empty body",
},
})
}
}), 20*time.Second), nil
}
// validBlobID ensures the ID is valid for a blob.
// The ID is always a UUID. As such, this checks for something that can resemble a UUID.
// This does not check for the ID to be an actual UUID, as the blob store may change their ID format, which we do not wish to stand in the way of.
func validBlobID(id string) bool {
for _, c := range id {
// [a-zA-Z0-9\-] are valid characters.
az := c >= 'a' && c <= 'z'
AZ := c >= 'A' && c <= 'Z'
digit := c >= '0' && c <= '9'
if !(az || AZ || digit || c == '-') {
return false
}
}
return true
}
var (
_ rest.Connecter = (*renderConnector)(nil)
_ rest.Storage = (*renderConnector)(nil)
_ rest.StorageMetadata = (*renderConnector)(nil)
)
pkg/registry/apis/provisioning/repository/context.go
+31
View File
@@ -0,0 +1,31 @@
package repository
import (
"context"
"time"
)
// When writing values from history, we want the metadata to match Legacy grafana SQL
type CommitSignature struct {
// Name represents a person name. It is an arbitrary string.
Name string
// Email is an email, but it cannot be assumed to be well-formed.
Email string
// When is the timestamp of the signature.
When time.Time
}
type ctxAuthorKey struct{}
func WithAuthorSignature(ctx context.Context, sig CommitSignature) context.Context {
return context.WithValue(ctx, ctxAuthorKey{}, sig)
}
func GetAuthorSignature(ctx context.Context) *CommitSignature {
u, ok := ctx.Value(ctxAuthorKey{}).(CommitSignature)
if ok {
copy := u
return &copy
}
return nil
}
pkg/registry/apis/provisioning/repository/context_test.go
+52
View File
@@ -0,0 +1,52 @@
package repository
import (
"context"
"testing"
"time"
"github.com/stretchr/testify/require"
)
func TestAuthorSignature(t *testing.T) {
t.Run("should store and retrieve author signature", func(t *testing.T) {
expected := CommitSignature{
Name: "John Doe",
Email: "john@example.com",
When: time.Date(2024, 1, 1, 12, 0, 0, 0, time.UTC),
}
ctx := context.Background()
ctx = WithAuthorSignature(ctx, expected)
result := GetAuthorSignature(ctx)
require.NotNil(t, result)
require.Equal(t, expected.Name, result.Name)
require.Equal(t, expected.Email, result.Email)
require.Equal(t, expected.When, result.When)
})
t.Run("should return nil when no signature is set", func(t *testing.T) {
ctx := context.Background()
result := GetAuthorSignature(ctx)
require.Nil(t, result)
})
t.Run("should return copy of signature", func(t *testing.T) {
original := CommitSignature{
Name: "John Doe",
Email: "john@example.com",
When: time.Now(),
}
ctx := context.Background()
ctx = WithAuthorSignature(ctx, original)
result1 := GetAuthorSignature(ctx)
result2 := GetAuthorSignature(ctx)
require.NotNil(t, result1)
require.NotNil(t, result2)
require.NotSame(t, result1, result2)
})
}
pkg/registry/apis/provisioning/repository/github.go
+916
View File
@@ -0,0 +1,916 @@
package repository
import (
"context"
"errors"
"fmt"
"log/slog"
"net/http"
"net/url"
"regexp"
"slices"
"strings"
"github.com/google/go-github/v69/github"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
"github.com/google/uuid"
"github.com/grafana/grafana-app-sdk/logging"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
pgh "github.com/grafana/grafana/pkg/registry/apis/provisioning/repository/github"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/secrets"
)
var subscribedEvents = []string{"push", "pull_request"}
// Make sure all public functions of this struct call the (*githubRepository).logger function, to ensure the GH repo details are included.
type githubRepository struct {
config *provisioning.Repository
gh pgh.Client // assumes github.com base URL
secrets secrets.Service
webhookURL string
owner string
repo string
}
var (
_ Repository = (*githubRepository)(nil)
_ Hooks = (*githubRepository)(nil)
_ Versioned = (*githubRepository)(nil)
_ Writer = (*githubRepository)(nil)
_ Reader = (*githubRepository)(nil)
_ RepositoryWithURLs = (*githubRepository)(nil)
)
func NewGitHub(
ctx context.Context,
config *provisioning.Repository,
factory *pgh.Factory,
secrets secrets.Service,
webhookURL string,
) (*githubRepository, error) {
owner, repo, err := parseOwnerRepo(config.Spec.GitHub.URL)
if err != nil {
return nil, err
}
token := config.Spec.GitHub.Token
if token == "" {
decrypted, err := secrets.Decrypt(ctx, config.Spec.GitHub.EncryptedToken)
if err != nil {
return nil, err
}
token = string(decrypted)
}
return &githubRepository{
config: config,
gh: factory.New(ctx, token), // TODO, baseURL from config
secrets: secrets,
webhookURL: webhookURL,
owner: owner,
repo: repo,
}, nil
}
func (r *githubRepository) Config() *provisioning.Repository {
return r.config
}
// Validate implements provisioning.Repository.
func (r *githubRepository) Validate() (list field.ErrorList) {
gh := r.config.Spec.GitHub
if gh == nil {
list = append(list, field.Required(field.NewPath("spec", "github"), "a github config is required"))
return list
}
if gh.URL == "" {
list = append(list, field.Required(field.NewPath("spec", "github", "url"), "a github url is required"))
} else {
_, _, err := parseOwnerRepo(gh.URL)
if err != nil {
list = append(list, field.Invalid(field.NewPath("spec", "github", "url"), gh.URL, err.Error()))
} else if !strings.HasPrefix(gh.URL, "https://github.com/") {
list = append(list, field.Invalid(field.NewPath("spec", "github", "url"), gh.URL, "URL must start with https://github.com/"))
}
}
if gh.Branch == "" {
list = append(list, field.Required(field.NewPath("spec", "github", "branch"), "a github branch is required"))
}
if !isValidGitBranchName(gh.Branch) {
list = append(list, field.Invalid(field.NewPath("spec", "github", "branch"), gh.Branch, "invalid branch name"))
}
// TODO: Use two fields for token
if gh.Token == "" && len(gh.EncryptedToken) == 0 {
list = append(list, field.Required(field.NewPath("spec", "github", "token"), "a github access token is required"))
}
if err := safepath.IsSafe(gh.Path); err != nil {
list = append(list, field.Invalid(field.NewPath("spec", "github", "prefix"), gh.Path, err.Error()))
}
if safepath.IsAbs(gh.Path) {
list = append(list, field.Invalid(field.NewPath("spec", "github", "prefix"), gh.Path, "path must be relative"))
}
return list
}
func parseOwnerRepo(giturl string) (owner string, repo string, err error) {
parsed, e := url.Parse(strings.TrimSuffix(giturl, ".git"))
if e != nil {
err = e
return
}
parts := strings.Split(parsed.Path, "/")
if len(parts) < 3 {
err = fmt.Errorf("unable to parse repo+owner from url")
return
}
return parts[1], parts[2], nil
}
func fromError(err error, code int) *provisioning.TestResults {
statusErr, ok := err.(apierrors.APIStatus)
if ok {
s := statusErr.Status()
return &provisioning.TestResults{
Code: int(s.Code),
Success: false,
Errors: []string{s.Message},
}
}
return &provisioning.TestResults{
Code: code,
Success: false,
Errors: []string{err.Error()},
}
}
// Test implements provisioning.Repository.
func (r *githubRepository) Test(ctx context.Context) (*provisioning.TestResults, error) {
if err := r.gh.IsAuthenticated(ctx); err != nil {
return fromError(err, http.StatusUnauthorized), nil
}
owner, repo, err := parseOwnerRepo(r.config.Spec.GitHub.URL)
if err != nil {
return fromError(err, http.StatusBadRequest), nil
}
// FIXME: check token permissions
ok, err := r.gh.RepoExists(ctx, owner, repo)
if err != nil {
return fromError(err, http.StatusBadRequest), nil
}
if !ok {
return &provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{"repository does not exist"},
}, nil
}
ok, err = r.gh.BranchExists(ctx, r.owner, r.repo, r.config.Spec.GitHub.Branch)
if err != nil {
return fromError(err, http.StatusBadRequest), nil
}
if !ok {
return &provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{"branch does not exist"},
}, nil
}
return &provisioning.TestResults{
Code: http.StatusOK,
Success: true,
}, nil
}
// ReadResource implements provisioning.Repository.
func (r *githubRepository) Read(ctx context.Context, filePath, ref string) (*FileInfo, error) {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
finalPath := safepath.Join(r.config.Spec.GitHub.Path, filePath)
content, dirContent, err := r.gh.GetContents(ctx, r.owner, r.repo, finalPath, ref)
if err != nil {
if errors.Is(err, pgh.ErrResourceNotFound) {
return nil, &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: fmt.Sprintf("file not found; path=%s ref=%s", finalPath, ref),
Code: http.StatusNotFound,
},
}
}
return nil, fmt.Errorf("get contents: %w", err)
}
if dirContent != nil {
return &FileInfo{
Path: filePath,
Ref: ref,
}, nil
}
data, err := content.GetFileContent()
if err != nil {
return nil, fmt.Errorf("get content: %w", err)
}
return &FileInfo{
Path: filePath,
Ref: ref,
Data: []byte(data),
Hash: content.GetSHA(),
}, nil
}
func (r *githubRepository) ReadTree(ctx context.Context, ref string) ([]FileTreeEntry, error) {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, logger := r.logger(ctx, ref)
tree, truncated, err := r.gh.GetTree(ctx, r.owner, r.repo, r.config.Spec.GitHub.Path, ref, true)
if err != nil {
if errors.Is(err, pgh.ErrResourceNotFound) {
return nil, &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: fmt.Sprintf("tree not found; ref=%s", ref),
Code: http.StatusNotFound,
},
}
}
}
if truncated {
logger.Warn("tree from github was truncated")
}
entries := make([]FileTreeEntry, 0, len(tree))
for _, entry := range tree {
converted := FileTreeEntry{
Path: entry.GetPath(),
Size: entry.GetSize(),
Hash: entry.GetSHA(),
Blob: !entry.IsDirectory(),
}
entries = append(entries, converted)
}
return entries, nil
}
func (r *githubRepository) Create(ctx context.Context, path, ref string, data []byte, comment string) error {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, _ = r.logger(ctx, ref)
if err := r.ensureBranchExists(ctx, ref); err != nil {
return fmt.Errorf("create branch on create: %w", err)
}
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
// Create .keep file if it is a directory
if safepath.IsDir(finalPath) {
if data != nil {
return apierrors.NewBadRequest("data cannot be provided for a directory")
}
finalPath = safepath.Join(finalPath, ".keep")
data = []byte{}
}
err := r.gh.CreateFile(ctx, r.owner, r.repo, finalPath, ref, comment, data)
if errors.Is(err, pgh.ErrResourceAlreadyExists) {
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "file already exists",
Code: http.StatusConflict,
},
}
}
return err
}
func (r *githubRepository) Update(ctx context.Context, path, ref string, data []byte, comment string) error {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, _ = r.logger(ctx, ref)
if err := r.ensureBranchExists(ctx, ref); err != nil {
return fmt.Errorf("create branch on update: %w", err)
}
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
file, _, err := r.gh.GetContents(ctx, r.owner, r.repo, finalPath, ref)
if err != nil {
if errors.Is(err, pgh.ErrResourceNotFound) {
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "file not found",
Code: http.StatusNotFound,
},
}
}
return fmt.Errorf("get content before file update: %w", err)
}
if file.IsDirectory() {
return apierrors.NewBadRequest("cannot update a directory")
}
if err := r.gh.UpdateFile(ctx, r.owner, r.repo, finalPath, ref, comment, file.GetSHA(), data); err != nil {
return fmt.Errorf("update file: %w", err)
}
return nil
}
func (r *githubRepository) Write(ctx context.Context, path string, ref string, data []byte, message string) error {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, _ = r.logger(ctx, ref)
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
return writeWithReadThenCreateOrUpdate(ctx, r, finalPath, ref, data, message)
}
func (r *githubRepository) Delete(ctx context.Context, path, ref, comment string) error {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, _ = r.logger(ctx, ref)
if err := r.ensureBranchExists(ctx, ref); err != nil {
return fmt.Errorf("create branch on delete: %w", err)
}
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
return r.deleteRecursively(ctx, finalPath, ref, comment)
}
func (r *githubRepository) deleteRecursively(ctx context.Context, path, ref, comment string) error {
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
file, contents, err := r.gh.GetContents(ctx, r.owner, r.repo, finalPath, ref)
if err != nil {
if errors.Is(err, pgh.ErrResourceNotFound) {
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "file not found",
Code: http.StatusNotFound,
},
}
}
return fmt.Errorf("finding file to delete: %w", err)
}
if file != nil && !file.IsDirectory() {
return r.gh.DeleteFile(ctx, r.owner, r.repo, finalPath, ref, comment, file.GetSHA())
}
for _, c := range contents {
if c.IsDirectory() {
if err := r.deleteRecursively(ctx, c.GetPath(), ref, comment); err != nil {
return fmt.Errorf("delete file recursive: %w", err)
}
continue
}
if err := r.gh.DeleteFile(ctx, r.owner, r.repo, c.GetPath(), ref, comment, c.GetSHA()); err != nil {
return fmt.Errorf("delete file: %w", err)
}
}
return nil
}
func (r *githubRepository) History(ctx context.Context, path, ref string) ([]provisioning.HistoryItem, error) {
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
ctx, _ = r.logger(ctx, ref)
finalPath := safepath.Join(r.config.Spec.GitHub.Path, path)
commits, err := r.gh.Commits(ctx, r.owner, r.repo, finalPath, ref)
if err != nil {
if errors.Is(err, pgh.ErrResourceNotFound) {
return nil, &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "path not found",
Code: http.StatusNotFound,
},
}
}
return nil, fmt.Errorf("get commits: %w", err)
}
ret := make([]provisioning.HistoryItem, 0, len(commits))
for _, commit := range commits {
authors := make([]provisioning.Author, 0)
if commit.Author != nil {
authors = append(authors, provisioning.Author{
Name: commit.Author.Name,
Username: commit.Author.Username,
AvatarURL: commit.Author.AvatarURL,
})
}
if commit.Committer != nil && commit.Author != nil && commit.Author.Name != commit.Committer.Name {
authors = append(authors, provisioning.Author{
Name: commit.Committer.Name,
Username: commit.Committer.Username,
AvatarURL: commit.Committer.AvatarURL,
})
}
ret = append(ret, provisioning.HistoryItem{
Ref: commit.Ref,
Message: commit.Message,
Authors: authors,
CreatedAt: commit.CreatedAt.UnixMilli(),
})
}
return ret, nil
}
// basicGitBranchNameRegex is a regular expression to validate a git branch name
// it does not cover all cases as positive lookaheads are not supported in Go's regexp
var basicGitBranchNameRegex = regexp.MustCompile(`^[a-zA-Z0-9\-\_\/\.]+$`)
// isValidGitBranchName checks if a branch name is valid.
// It uses the following regexp `^[a-zA-Z0-9\-\_\/\.]+$` to validate the branch name with some additional checks that must satisfy the following rules:
// 1. The branch name must have at least one character and must not be empty.
// 2. The branch name cannot start with `/` or end with `/`, `.`, or whitespace.
// 3. The branch name cannot contain consecutive slashes (`//`).
// 4. The branch name cannot contain consecutive dots (`..`).
// 5. The branch name cannot contain `@{`.
// 6. The branch name cannot include the following characters: `~`, `^`, `:`, `?`, `*`, `[`, `\`, or `]`.
func isValidGitBranchName(branch string) bool {
if !basicGitBranchNameRegex.MatchString(branch) {
return false
}
// Additional checks for invalid patterns
if strings.HasPrefix(branch, "/") || strings.HasSuffix(branch, "/") ||
strings.HasSuffix(branch, ".") || strings.Contains(branch, "..") ||
strings.Contains(branch, "//") || strings.HasSuffix(branch, ".lock") {
return false
}
return true
}
func (r *githubRepository) ensureBranchExists(ctx context.Context, branchName string) error {
if !isValidGitBranchName(branchName) {
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Code: http.StatusBadRequest,
Message: "invalid branch name",
},
}
}
ok, err := r.gh.BranchExists(ctx, r.owner, r.repo, branchName)
if err != nil {
return fmt.Errorf("check branch exists: %w", err)
}
if ok {
logging.FromContext(ctx).Info("branch already exists", "branch", branchName)
return nil
}
srcBranch := r.config.Spec.GitHub.Branch
if err := r.gh.CreateBranch(ctx, r.owner, r.repo, srcBranch, branchName); err != nil {
if errors.Is(err, pgh.ErrResourceAlreadyExists) {
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Code: http.StatusConflict,
Message: "branch already exists",
},
}
}
return fmt.Errorf("create branch: %w", err)
}
return nil
}
// Webhook implements Repository.
func (r *githubRepository) Webhook(ctx context.Context, req *http.Request) (*provisioning.WebhookResponse, error) {
if r.config.Status.Webhook == nil {
return nil, fmt.Errorf("unexpected webhook request")
}
secret, err := r.secrets.Decrypt(ctx, r.config.Status.Webhook.EncryptedSecret)
if err != nil {
return nil, fmt.Errorf("failed to decrypt secret: %w", err)
}
payload, err := github.ValidatePayload(req, secret)
if err != nil {
return nil, apierrors.NewUnauthorized("invalid signature")
}
return r.parseWebhook(github.WebHookType(req), payload)
}
// This method does not include context because it does delegate any more requests
func (r *githubRepository) parseWebhook(messageType string, payload []byte) (*provisioning.WebhookResponse, error) {
event, err := github.ParseWebHook(messageType, payload)
if err != nil {
return nil, apierrors.NewBadRequest("invalid payload")
}
switch event := event.(type) {
case *github.PushEvent:
return r.parsePushEvent(event)
case *github.PullRequestEvent:
return r.parsePullRequestEvent(event)
case *github.PingEvent:
return &provisioning.WebhookResponse{
Code: http.StatusOK,
Message: "ping received",
}, nil
}
return &provisioning.WebhookResponse{
Code: http.StatusNotImplemented,
Message: fmt.Sprintf("unsupported messageType: %s", messageType),
}, nil
}
func (r *githubRepository) parsePushEvent(event *github.PushEvent) (*provisioning.WebhookResponse, error) {
if event.GetRepo() == nil {
return nil, fmt.Errorf("missing repository in push event")
}
if event.GetRepo().GetFullName() != fmt.Sprintf("%s/%s", r.owner, r.repo) {
return nil, fmt.Errorf("repository mismatch")
}
// No need to sync if not enabled
if !r.config.Spec.Sync.Enabled {
return &provisioning.WebhookResponse{Code: http.StatusOK}, nil
}
// Skip silently if the event is not for the main/master branch
// as we cannot configure the webhook to only publish events for the main branch
if event.GetRef() != fmt.Sprintf("refs/heads/%s", r.config.Spec.GitHub.Branch) {
return &provisioning.WebhookResponse{Code: http.StatusOK}, nil
}
return &provisioning.WebhookResponse{
Code: http.StatusAccepted,
Job: &provisioning.JobSpec{
Repository: r.Config().GetName(),
Action: provisioning.JobActionSync,
Pull: &provisioning.SyncJobOptions{
Incremental: true,
},
},
}, nil
}
func (r *githubRepository) parsePullRequestEvent(event *github.PullRequestEvent) (*provisioning.WebhookResponse, error) {
if event.GetRepo() == nil {
return nil, fmt.Errorf("missing repository in pull request event")
}
cfg := r.config.Spec.GitHub
if cfg == nil {
return nil, fmt.Errorf("missing github config")
}
if event.GetRepo().GetFullName() != fmt.Sprintf("%s/%s", r.owner, r.repo) {
return nil, fmt.Errorf("repository mismatch")
}
pr := event.GetPullRequest()
if pr == nil {
return nil, fmt.Errorf("expected PR in event")
}
if pr.GetBase().GetRef() != r.config.Spec.GitHub.Branch {
return &provisioning.WebhookResponse{
Code: http.StatusOK,
Message: fmt.Sprintf("ignoring pull request event as %s is not the configured branch", pr.GetBase().GetRef()),
}, nil
}
action := event.GetAction()
if action != "opened" && action != "reopened" && action != "synchronize" {
return &provisioning.WebhookResponse{
Code: http.StatusOK, // Nothing needed
Message: fmt.Sprintf("ignore pull request event: %s", action),
}, nil
}
// Queue an async job that will parse files
return &provisioning.WebhookResponse{
Code: http.StatusAccepted, // Nothing needed
Message: fmt.Sprintf("pull request: %s", action),
Job: &provisioning.JobSpec{
Repository: r.Config().GetName(),
Action: provisioning.JobActionPullRequest,
PullRequest: &provisioning.PullRequestJobOptions{
URL: pr.GetHTMLURL(),
PR: pr.GetNumber(),
Ref: pr.GetHead().GetRef(),
Hash: pr.GetHead().GetSHA(),
},
},
}, nil
}
func (r *githubRepository) LatestRef(ctx context.Context) (string, error) {
ctx, _ = r.logger(ctx, "")
branch, err := r.gh.GetBranch(ctx, r.owner, r.repo, r.Config().Spec.GitHub.Branch)
if err != nil {
return "", fmt.Errorf("get branch: %w", err)
}
return branch.Sha, nil
}
func (r *githubRepository) CompareFiles(ctx context.Context, base, ref string) ([]VersionedFileChange, error) {
if ref == "" {
var err error
ref, err = r.LatestRef(ctx)
if err != nil {
return nil, fmt.Errorf("get latest ref: %w", err)
}
}
ctx, logger := r.logger(ctx, ref)
files, err := r.gh.CompareCommits(ctx, r.owner, r.repo, base, ref)
if err != nil {
return nil, fmt.Errorf("compare commits: %w", err)
}
changes := make([]VersionedFileChange, 0)
for _, f := range files {
// reference: https://docs.github.com/en/rest/commits/commits?apiVersion=2022-11-28#get-a-commit
switch f.GetStatus() {
case "added", "copied":
changes = append(changes, VersionedFileChange{
Path: f.GetFilename(),
Ref: ref,
Action: FileActionCreated,
})
case "modified", "changed":
changes = append(changes, VersionedFileChange{
Path: f.GetFilename(),
Ref: ref,
Action: FileActionUpdated,
})
case "renamed":
changes = append(changes, VersionedFileChange{
Path: f.GetFilename(),
PreviousPath: f.GetPreviousFilename(),
Ref: ref,
PreviousRef: base,
Action: FileActionRenamed,
})
case "removed":
changes = append(changes, VersionedFileChange{
Ref: base,
Path: f.GetFilename(),
Action: FileActionDeleted,
})
case "unchanged":
// do nothing
default:
logger.Error("ignore unhandled file", "file", f.GetFilename(), "status", f.GetStatus())
}
}
return changes, nil
}
// ClearAllPullRequestFileComments clears all comments on a pull request
func (r *githubRepository) ClearAllPullRequestFileComments(ctx context.Context, prNumber int) error {
ctx, _ = r.logger(ctx, "")
return r.gh.ClearAllPullRequestFileComments(ctx, r.owner, r.repo, prNumber)
}
// CommentPullRequest adds a comment to a pull request.
func (r *githubRepository) CommentPullRequest(ctx context.Context, prNumber int, comment string) error {
ctx, _ = r.logger(ctx, "")
return r.gh.CreatePullRequestComment(ctx, r.owner, r.repo, prNumber, comment)
}
// CommentPullRequestFile lints a file and comments the issues found.
func (r *githubRepository) CommentPullRequestFile(ctx context.Context, prNumber int, path, ref, comment string) error {
ctx, _ = r.logger(ctx, ref)
fileComment := pgh.FileComment{
Content: comment,
Path: path,
Position: 1, // create a top-level comment
Ref: ref,
}
// FIXME: comment with Grafana Logo
// FIXME: comment author should be written by Grafana and not the user
return r.gh.CreatePullRequestFileComment(ctx, r.owner, r.repo, prNumber, fileComment)
}
// ResourceURLs implements RepositoryWithURLs.
func (r *githubRepository) ResourceURLs(ctx context.Context, file *FileInfo) (*provisioning.ResourceURLs, error) {
cfg := r.config.Spec.GitHub
if file.Path == "" || cfg == nil {
return nil, nil
}
ref := file.Ref
if ref == "" {
ref = cfg.Branch
}
urls := &provisioning.ResourceURLs{
RepositoryURL: cfg.URL,
SourceURL: fmt.Sprintf("%s/blob/%s/%s", cfg.URL, ref, file.Path),
}
if ref != cfg.Branch {
urls.CompareURL = fmt.Sprintf("%s/compare/%s...%s", cfg.URL, cfg.Branch, ref)
// Create a new pull request
urls.NewPullRequestURL = fmt.Sprintf("%s?quick_pull=1&labels=grafana", urls.CompareURL)
}
return urls, nil
}
func (r *githubRepository) createWebhook(ctx context.Context) (pgh.WebhookConfig, error) {
secret, err := uuid.NewRandom()
if err != nil {
return pgh.WebhookConfig{}, fmt.Errorf("could not generate secret: %w", err)
}
cfg := pgh.WebhookConfig{
URL: r.webhookURL,
Secret: secret.String(),
ContentType: "json",
Events: subscribedEvents,
Active: true,
}
hook, err := r.gh.CreateWebhook(ctx, r.owner, r.repo, cfg)
if err != nil {
return pgh.WebhookConfig{}, err
}
// HACK: GitHub does not return the secret, so we need to update it manually
hook.Secret = cfg.Secret
logging.FromContext(ctx).Info("webhook created", "url", cfg.URL, "id", hook.ID)
return hook, nil
}
// updateWebhook checks if the webhook needs to be updated and updates it if necessary.
// if the webhook does not exist, it will create it.
func (r *githubRepository) updateWebhook(ctx context.Context) (pgh.WebhookConfig, bool, error) {
if r.config.Status.Webhook == nil || r.config.Status.Webhook.ID == 0 {
hook, err := r.createWebhook(ctx)
if err != nil {
return pgh.WebhookConfig{}, false, err
}
return hook, true, nil
}
hook, err := r.gh.GetWebhook(ctx, r.owner, r.repo, r.config.Status.Webhook.ID)
switch {
case errors.Is(err, pgh.ErrResourceNotFound):
hook, err := r.createWebhook(ctx)
if err != nil {
return pgh.WebhookConfig{}, false, err
}
return hook, true, nil
case err != nil:
return pgh.WebhookConfig{}, false, fmt.Errorf("get webhook: %w", err)
}
hook.Secret = r.config.Status.Webhook.Secret // we always random gen this, so don't use it for mustUpdate below.
var mustUpdate bool
if hook.URL != r.config.Status.Webhook.URL {
mustUpdate = true
hook.URL = r.webhookURL
}
if !slices.Equal(hook.Events, subscribedEvents) {
mustUpdate = true
hook.Events = subscribedEvents
}
if !mustUpdate {
return hook, false, nil
}
// Something has changed in the webhook. Let's rotate the secret as well, so as to ensure we end up with a 100% correct webhook.
secret, err := uuid.NewRandom()
if err != nil {
return pgh.WebhookConfig{}, false, fmt.Errorf("could not generate secret: %w", err)
}
hook.Secret = secret.String()
if err := r.gh.EditWebhook(ctx, r.owner, r.repo, hook); err != nil {
return pgh.WebhookConfig{}, false, fmt.Errorf("edit webhook: %w", err)
}
return hook, true, nil
}
func (r *githubRepository) deleteWebhook(ctx context.Context) error {
if r.config.Status.Webhook == nil {
return fmt.Errorf("webhook not found")
}
id := r.config.Status.Webhook.ID
if err := r.gh.DeleteWebhook(ctx, r.owner, r.repo, id); err != nil {
return fmt.Errorf("delete webhook: %w", err)
}
logging.FromContext(ctx).Info("webhook deleted", "url", r.config.Status.Webhook.URL, "id", id)
return nil
}
func (r *githubRepository) OnCreate(ctx context.Context) (*provisioning.WebhookStatus, error) {
if len(r.webhookURL) == 0 {
return nil, nil
}
ctx, _ = r.logger(ctx, "")
hook, err := r.createWebhook(ctx)
if err != nil {
return nil, err
}
return &provisioning.WebhookStatus{
ID: hook.ID,
URL: hook.URL,
Secret: hook.Secret,
SubscribedEvents: hook.Events,
}, nil
}
func (r *githubRepository) OnUpdate(ctx context.Context) (*provisioning.WebhookStatus, error) {
if len(r.webhookURL) == 0 {
return nil, nil
}
ctx, _ = r.logger(ctx, "")
hook, _, err := r.updateWebhook(ctx)
if err != nil {
return nil, err
}
return &provisioning.WebhookStatus{
ID: hook.ID,
URL: hook.URL,
Secret: hook.Secret,
SubscribedEvents: hook.Events,
}, nil
}
func (r *githubRepository) OnDelete(ctx context.Context) error {
if len(r.webhookURL) == 0 {
return nil
}
ctx, _ = r.logger(ctx, "")
return r.deleteWebhook(ctx)
}
func (r *githubRepository) logger(ctx context.Context, ref string) (context.Context, logging.Logger) {
logger := logging.FromContext(ctx)
type containsGh int
var containsGhKey containsGh
if ctx.Value(containsGhKey) != nil {
return ctx, logging.FromContext(ctx)
}
if ref == "" {
ref = r.config.Spec.GitHub.Branch
}
logger = logger.With(slog.Group("github_repository", "owner", r.owner, "name", r.repo, "ref", ref))
ctx = logging.Context(ctx, logger)
// We want to ensure we don't add multiple github_repository keys. With doesn't deduplicate the keys...
ctx = context.WithValue(ctx, containsGhKey, true)
return ctx, logger
}
pkg/registry/apis/provisioning/repository/github/client.go
+182
View File
@@ -0,0 +1,182 @@
// The github package exists to provide a client for the GH API, which can also be faked with a mock.
// In most cases, we want the real client, but testing should mock it, lest we get blocked from their API, or have to configure auth for simple tests.
package github
import (
"context"
"errors"
"time"
"github.com/google/go-github/v69/github"
apierrors "k8s.io/apimachinery/pkg/api/errors"
)
// API errors that we need to convey after parsing real GH errors (or faking them).
var (
ErrResourceAlreadyExists = errors.New("the resource already exists")
ErrResourceNotFound = errors.New("the resource does not exist")
ErrMismatchedHash = errors.New("the update cannot be applied because the expected and actual hashes are unequal")
ErrNoSecret = errors.New("new webhooks must have a secret")
//lint:ignore ST1005 this is not punctuation
ErrPathTraversalDisallowed = errors.New("the path contained ..") //nolint:stylecheck
ErrServiceUnavailable = apierrors.NewServiceUnavailable("github is unavailable")
ErrFileTooLarge = errors.New("file exceeds maximum allowed size")
ErrTooManyItems = errors.New("maximum number of items exceeded")
)
// MaxFileSize maximum file size limit (10MB)
const MaxFileSize = 10 * 1024 * 1024 // 10MB in bytes
type ErrRateLimited = github.RateLimitError
type Client interface {
// IsAuthenticated checks if the client is authenticated.
IsAuthenticated(ctx context.Context) error
// GetContents returns the metadata and content of a file or directory.
// When a file is checked, the first returned value will have a value. For a directory, the second will. The other value is always nil.
// If an error occurs, the returned values may or may not be nil.
//
// If ".." appears in the "path", this method will return an error.
GetContents(ctx context.Context, owner, repository, path, ref string) (fileContents RepositoryContent, dirContents []RepositoryContent, err error)
// GetTree returns the Git tree in the repository.
// When recursive is given, subtrees are mapped into the returned array.
// When basePath is given, only trees under it are given. The results do not include this path in their names.
//
// The truncated bool will be set to true if the tree is larger than 7 MB or 100 000 entries.
// When truncated is true, you may wish to read each subtree manually instead.
GetTree(ctx context.Context, owner, repository, basePath, ref string, recursive bool) (entries []RepositoryContent, truncated bool, err error)
// CreateFile creates a new file in the repository under the given path.
// The file is created on the branch given.
// The message given is the commit message. If none is given, an appropriate default is used.
// The content is what the file should contain. An empty slice is valid, though often not very useful.
//
// If ".." appears in the "path", this method will return an error.
CreateFile(ctx context.Context, owner, repository, path, branch, message string, content []byte) error
// UpdateFile updates a file in the repository under the given path.
// The file is updated on the branch given.
// The message given is the commit message. If none is given, an appropriate default is used.
// The content is what the file should contain. An empty slice is valid, though often not very useful.
// If the path does not exist, an error is returned.
// The hash given must be the SHA hash of the file contents. Calling GetContents in advance is an easy way of handling this.
//
// If ".." appears in the "path", this method will return an error.
UpdateFile(ctx context.Context, owner, repository, path, branch, message, hash string, content []byte) error
// DeleteFile deletes a file in the repository under the given path.
// The file is deleted from the branch given.
// The message given is the commit message. If none is given, an appropriate default is used.
// If the path does not exist, an error is returned.
// The hash given must be the SHA hash of the file contents. Calling GetContents in advance is an easy way of handling this.
//
// If ".." appears in the "path", this method will return an error.
DeleteFile(ctx context.Context, owner, repository, path, branch, message, hash string) error
// Commits returns the commits for the given path
Commits(ctx context.Context, owner, repository, path, branch string) ([]Commit, error)
// CompareCommits returns the changes between two commits.
CompareCommits(ctx context.Context, owner, repository, base, head string) ([]CommitFile, error)
// RepoExists checks if a repository exists.
RepoExists(ctx context.Context, owner, repository string) (bool, error)
// CreateBranch creates a new branch in the repository.
CreateBranch(ctx context.Context, owner, repository, sourceBranch, branchName string) error
// BranchExists checks if a branch exists in the repository.
BranchExists(ctx context.Context, owner, repository, branchName string) (bool, error)
// GetBranch returns the branch of the repository.
GetBranch(ctx context.Context, owner, repository, branchName string) (Branch, error)
ListWebhooks(ctx context.Context, owner, repository string) ([]WebhookConfig, error)
CreateWebhook(ctx context.Context, owner, repository string, cfg WebhookConfig) (WebhookConfig, error)
GetWebhook(ctx context.Context, owner, repository string, webhookID int64) (WebhookConfig, error)
DeleteWebhook(ctx context.Context, owner, repository string, webhookID int64) error
EditWebhook(ctx context.Context, owner, repository string, cfg WebhookConfig) error
ListPullRequestFiles(ctx context.Context, owner, repository string, number int) ([]CommitFile, error)
CreatePullRequestComment(ctx context.Context, owner, repository string, number int, body string) error
CreatePullRequestFileComment(ctx context.Context, owner, repository string, number int, comment FileComment) error
ClearAllPullRequestFileComments(ctx context.Context, owner, repository string, number int) error
}
type RepositoryContent interface {
// Returns true if this is a directory, false if it is a file.
IsDirectory() bool
// Returns the contents of the file. Decoding happens if necessary.
// Returns an error if the content represents a directory.
GetFileContent() (string, error)
// Returns true if this is a symlink.
// If true, GetPath returns the path where this symlink leads.
IsSymlink() bool
// Returns the full path from the root of the repository.
// This has no leading or trailing slashes.
// The path only uses '/' for directories. You can use the 'path' package to interact with these.
GetPath() string
// Get the SHA hash. This is usually a SHA-256, but may also be SHA-512.
// Directories have SHA hashes, too (TODO: how is this calculated?).
GetSHA() string
// The size of the file. Not necessarily non-zero, even if the file is supposed to be non-zero.
GetSize() int64
}
type Branch struct {
Name string
Sha string
}
type CommitAuthor struct {
Name string
Username string
AvatarURL string
}
type Commit struct {
Ref string
Message string
Author *CommitAuthor
Committer *CommitAuthor
CreatedAt time.Time
}
type CommitFile interface {
GetSHA() string
GetFilename() string
GetPreviousFilename() string
GetStatus() string
}
type FileComment struct {
Content string
Path string
Position int
Ref string
}
type CreateFileOptions struct {
// The message of the commit. May be empty, in which case a default value is entered.
Message string
// The content of the file to write, unencoded.
Content []byte
}
type WebhookConfig struct {
// The ID of the webhook.
// Can be 0 on creation.
ID int64
// The events which this webhook shall contact the URL for.
Events []string
// Is the webhook enabled?
Active bool
// The URL GitHub should contact on events.
URL string
// The content type GitHub should send to the URL.
// If not specified, this is "form".
ContentType string
// The secret to use when sending events to the URL.
// If fetched from GitHub, this is empty as it contains no useful information.
Secret string
}
pkg/registry/apis/provisioning/repository/github/factory.go
+32
View File
@@ -0,0 +1,32 @@
package github
import (
"context"
"net/http"
"github.com/google/go-github/v69/github"
"golang.org/x/oauth2"
)
// Factory creates new GitHub clients.
// It exists only for the ability to test the code easily.
type Factory struct {
// Client allows overriding the client to use in the GH client returned. It exists primarily for testing.
Client *http.Client
}
func ProvideFactory() *Factory {
return &Factory{}
}
func (r *Factory) New(ctx context.Context, ghToken string) Client {
if r.Client != nil {
return NewClient(github.NewClient(r.Client))
}
tokenSrc := oauth2.StaticTokenSource(
&oauth2.Token{AccessToken: ghToken},
)
tokenClient := oauth2.NewClient(ctx, tokenSrc)
return NewClient(github.NewClient(tokenClient))
}
pkg/registry/apis/provisioning/repository/github/impl.go
+770
View File
@@ -0,0 +1,770 @@
package github
import (
"context"
"errors"
"fmt"
"net/http"
"time"
"github.com/google/go-github/v69/github"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type githubClient struct {
gh *github.Client
}
var _ Client = (*githubClient)(nil)
func NewClient(client *github.Client) *githubClient {
return &githubClient{client}
}
func (r *githubClient) IsAuthenticated(ctx context.Context) error {
if _, _, err := r.gh.Users.Get(ctx, ""); err != nil {
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) {
switch ghErr.Response.StatusCode {
case http.StatusUnauthorized:
return apierrors.NewUnauthorized("token is invalid or expired")
case http.StatusForbidden:
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Status: metav1.StatusFailure,
Code: http.StatusUnauthorized,
Reason: metav1.StatusReasonUnauthorized,
Message: "token is revoked or has insufficient permissions",
},
}
case http.StatusServiceUnavailable:
return ErrServiceUnavailable
}
}
return err
}
return nil
}
func (r *githubClient) RepoExists(ctx context.Context, owner, repository string) (bool, error) {
_, resp, err := r.gh.Repositories.Get(ctx, owner, repository)
if err == nil {
return true, nil
}
if resp.StatusCode == http.StatusNotFound {
return false, nil
}
return false, err
}
const (
maxDirectoryItems = 1000 // Maximum number of items allowed in a directory
maxTreeItems = 10000 // Maximum number of items allowed in a tree
maxCommits = 1000 // Maximum number of commits to fetch
maxCompareFiles = 1000 // Maximum number of files to compare between commits
maxWebhooks = 100 // Maximum number of webhooks allowed per repository
maxPRFiles = 1000 // Maximum number of files allowed in a pull request
maxPullRequestsFileComments = 1000 // Maximum number of comments allowed in a pull request
maxFileSize = 10 * 1024 * 1024 // 10MB in bytes
)
func (r *githubClient) GetContents(ctx context.Context, owner, repository, path, ref string) (fileContents RepositoryContent, dirContents []RepositoryContent, err error) {
// First try to get repository contents
opts := &github.RepositoryContentGetOptions{
Ref: ref,
}
fc, dc, _, err := r.gh.Repositories.GetContents(ctx, owner, repository, path, opts)
if err != nil {
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return nil, nil, err
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return nil, nil, ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return nil, nil, ErrResourceNotFound
}
return nil, nil, err
}
if fc != nil {
// Check file size before returning content
if fc.GetSize() > maxFileSize {
return nil, nil, ErrFileTooLarge
}
return realRepositoryContent{fc}, nil, nil
}
// For directories, check size limits
if len(dc) > maxDirectoryItems {
return nil, nil, fmt.Errorf("directory contains too many items (more than %d)", maxDirectoryItems)
}
// Convert directory contents
allContents := make([]RepositoryContent, 0, len(dc))
for _, original := range dc {
allContents = append(allContents, realRepositoryContent{original})
}
return nil, allContents, nil
}
func (r *githubClient) GetTree(ctx context.Context, owner, repository, basePath, ref string, recursive bool) ([]RepositoryContent, bool, error) {
var tree *github.Tree
var err error
subPaths := safepath.Split(basePath)
currentRef := ref
for {
// If subPaths is empty, we can read recursively, as we're reading the tree from the "base" of the repository. Otherwise, always read only the direct children.
recursive := recursive && len(subPaths) == 0
tree, _, err = r.gh.Git.GetTree(ctx, owner, repository, currentRef, recursive)
if err != nil {
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return nil, false, err
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return nil, false, ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
if currentRef != ref {
// We're operating with a subpath which doesn't exist yet.
// Pretend as if there is simply no files.
return nil, false, nil
}
// currentRef == ref
// This indicates the repository or commitish reference doesn't exist. This should always return an error.
return nil, false, ErrResourceNotFound
}
return nil, false, err
}
// Check if we've exceeded the maximum allowed items
if len(tree.Entries) > maxTreeItems {
return nil, false, fmt.Errorf("tree contains too many items (more than %d)", maxTreeItems)
}
// Prep for next iteration.
if len(subPaths) == 0 {
// We're done: we've discovered the tree we want.
break
}
// the ref must be equal the SHA of the entry corresponding to subPaths[0]
currentRef = ""
for _, e := range tree.Entries {
if e.GetPath() == subPaths[0] {
currentRef = e.GetSHA()
break
}
}
subPaths = subPaths[1:]
if currentRef == "" {
// We couldn't find the folder in the tree...
return nil, false, nil
}
}
// If the tree is truncated and we're in recursive mode, return an error
if tree.GetTruncated() && recursive {
return nil, true, fmt.Errorf("tree is too large to fetch recursively (more than %d items)", maxTreeItems)
}
entries := make([]RepositoryContent, 0, len(tree.Entries))
for _, te := range tree.Entries {
rrc := &realRepositoryContent{
real: &github.RepositoryContent{
Path: te.Path,
Size: te.Size,
SHA: te.SHA,
},
}
if te.GetType() == "tree" {
rrc.real.Type = github.Ptr("dir")
} else {
rrc.real.Type = te.Type
}
entries = append(entries, rrc)
}
return entries, tree.GetTruncated(), nil
}
func (r *githubClient) CreateFile(ctx context.Context, owner, repository, path, branch, message string, content []byte) error {
if message == "" {
message = fmt.Sprintf("Create %s", path)
}
_, _, err := r.gh.Repositories.CreateFile(ctx, owner, repository, path, &github.RepositoryContentFileOptions{
Branch: &branch,
Message: &message,
Content: content,
})
if err == nil {
return nil
}
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return err
}
if ghErr.Response.StatusCode == http.StatusUnprocessableEntity {
return ErrResourceAlreadyExists
}
return err
}
func (r *githubClient) UpdateFile(ctx context.Context, owner, repository, path, branch, message, hash string, content []byte) error {
if message == "" {
message = fmt.Sprintf("Update %s", path)
}
_, _, err := r.gh.Repositories.UpdateFile(ctx, owner, repository, path, &github.RepositoryContentFileOptions{
Branch: &branch,
Message: &message,
Content: content,
SHA: &hash,
})
if err == nil {
return nil
}
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return err
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return ErrResourceNotFound
}
if ghErr.Response.StatusCode == http.StatusConflict {
return ErrMismatchedHash
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
return err
}
func (r *githubClient) DeleteFile(ctx context.Context, owner, repository, path, branch, message, hash string) error {
if message == "" {
message = fmt.Sprintf("Delete %s", path)
}
_, _, err := r.gh.Repositories.DeleteFile(ctx, owner, repository, path, &github.RepositoryContentFileOptions{
Branch: &branch,
Message: &message,
SHA: &hash,
})
if err == nil {
return nil
}
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return err
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return ErrResourceNotFound
}
if ghErr.Response.StatusCode == http.StatusConflict {
return ErrMismatchedHash
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
return err
}
// Commits returns a list of commits for a given repository and branch.
func (r *githubClient) Commits(ctx context.Context, owner, repository, path, branch string) ([]Commit, error) {
listFn := func(ctx context.Context, opts *github.ListOptions) ([]*github.RepositoryCommit, *github.Response, error) {
return r.gh.Repositories.ListCommits(ctx, owner, repository, &github.CommitsListOptions{
Path: path,
SHA: branch,
ListOptions: *opts,
})
}
commits, err := paginatedList(
ctx,
listFn,
defaultListOptions(maxCommits),
)
if errors.Is(err, ErrTooManyItems) {
return nil, fmt.Errorf("too many commits to fetch (more than %d)", maxCommits)
}
if err != nil {
return nil, err
}
ret := make([]Commit, 0, len(commits))
for _, c := range commits {
var createdAt time.Time
var author *CommitAuthor
if c.GetCommit().GetAuthor() != nil {
author = &CommitAuthor{
Name: c.GetCommit().GetAuthor().GetName(),
Username: c.GetAuthor().GetLogin(),
AvatarURL: c.GetAuthor().GetAvatarURL(),
}
createdAt = c.GetCommit().GetAuthor().GetDate().Time
}
var committer *CommitAuthor
if c.GetCommitter() != nil {
committer = &CommitAuthor{
Name: c.GetCommit().GetCommitter().GetName(),
Username: c.GetCommitter().GetLogin(),
AvatarURL: c.GetCommitter().GetAvatarURL(),
}
}
ret = append(ret, Commit{
Ref: c.GetSHA(),
Message: c.GetCommit().GetMessage(),
Author: author,
Committer: committer,
CreatedAt: createdAt,
})
}
return ret, nil
}
func (r *githubClient) CompareCommits(ctx context.Context, owner, repository, base, head string) ([]CommitFile, error) {
listFn := func(ctx context.Context, opts *github.ListOptions) ([]*github.CommitFile, *github.Response, error) {
compare, resp, err := r.gh.Repositories.CompareCommits(ctx, owner, repository, base, head, opts)
if err != nil {
return nil, resp, err
}
return compare.Files, resp, nil
}
files, err := paginatedList(
ctx,
listFn,
defaultListOptions(maxCompareFiles),
)
if errors.Is(err, ErrTooManyItems) {
return nil, fmt.Errorf("too many files changed between commits (more than %d)", maxCompareFiles)
}
if err != nil {
return nil, err
}
// Convert to the interface type
ret := make([]CommitFile, 0, len(files))
for _, f := range files {
ret = append(ret, f)
}
return ret, nil
}
func (r *githubClient) GetBranch(ctx context.Context, owner, repository, branchName string) (Branch, error) {
branch, _, err := r.gh.Repositories.GetBranch(ctx, owner, repository, branchName, 0)
if err != nil {
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return Branch{}, err
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return Branch{}, ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return Branch{}, ErrResourceNotFound
}
return Branch{}, err
}
return Branch{
Name: branch.GetName(),
Sha: branch.GetCommit().GetSHA(),
}, nil
}
func (r *githubClient) CreateBranch(ctx context.Context, owner, repository, sourceBranch, branchName string) error {
// Fail if the branch already exists
if _, _, err := r.gh.Repositories.GetBranch(ctx, owner, repository, branchName, 0); err == nil {
return ErrResourceAlreadyExists
}
// Branch out based on the repository branch
baseRef, _, err := r.gh.Repositories.GetBranch(ctx, owner, repository, sourceBranch, 0)
if err != nil {
return fmt.Errorf("get base branch: %w", err)
}
if _, _, err := r.gh.Git.CreateRef(ctx, owner, repository, &github.Reference{
Ref: github.Ptr(fmt.Sprintf("refs/heads/%s", branchName)),
Object: &github.GitObject{
SHA: baseRef.Commit.SHA,
},
}); err != nil {
return fmt.Errorf("create branch ref: %w", err)
}
return nil
}
func (r *githubClient) BranchExists(ctx context.Context, owner, repository, branchName string) (bool, error) {
_, resp, err := r.gh.Repositories.GetBranch(ctx, owner, repository, branchName, 0)
if err == nil {
return true, nil
}
if resp.StatusCode == http.StatusNotFound {
return false, nil
}
return false, err
}
func (r *githubClient) ListWebhooks(ctx context.Context, owner, repository string) ([]WebhookConfig, error) {
listFn := func(ctx context.Context, opts *github.ListOptions) ([]*github.Hook, *github.Response, error) {
return r.gh.Repositories.ListHooks(ctx, owner, repository, opts)
}
hooks, err := paginatedList(
ctx,
listFn,
defaultListOptions(maxWebhooks),
)
if errors.Is(err, ErrTooManyItems) {
return nil, fmt.Errorf("too many webhooks configured (more than %d)", maxWebhooks)
}
if err != nil {
return nil, err
}
// Pre-allocate the result slice
ret := make([]WebhookConfig, 0, len(hooks))
for _, h := range hooks {
contentType := h.GetConfig().GetContentType()
if contentType == "" {
contentType = "form"
}
ret = append(ret, WebhookConfig{
ID: h.GetID(),
Events: h.Events,
Active: h.GetActive(),
URL: h.GetConfig().GetURL(),
ContentType: contentType,
// Intentionally not setting Secret.
})
}
return ret, nil
}
func (r *githubClient) CreateWebhook(ctx context.Context, owner, repository string, cfg WebhookConfig) (WebhookConfig, error) {
if cfg.ContentType == "" {
cfg.ContentType = "form"
}
hook := &github.Hook{
URL: &cfg.URL,
Events: cfg.Events,
Active: &cfg.Active,
Config: &github.HookConfig{
ContentType: &cfg.ContentType,
Secret: &cfg.Secret,
URL: &cfg.URL,
},
}
createdHook, _, err := r.gh.Repositories.CreateHook(ctx, owner, repository, hook)
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) && ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return WebhookConfig{}, ErrServiceUnavailable
}
if err != nil {
return WebhookConfig{}, err
}
return WebhookConfig{
ID: createdHook.GetID(),
// events is not returned by GitHub.
Events: cfg.Events,
Active: createdHook.GetActive(),
URL: createdHook.GetConfig().GetURL(),
ContentType: createdHook.GetConfig().GetContentType(),
// Secret is not returned by GitHub.
Secret: cfg.Secret,
}, nil
}
func (r *githubClient) GetWebhook(ctx context.Context, owner, repository string, webhookID int64) (WebhookConfig, error) {
hook, _, err := r.gh.Repositories.GetHook(ctx, owner, repository, webhookID)
if err != nil {
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) && ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return WebhookConfig{}, ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return WebhookConfig{}, ErrResourceNotFound
}
return WebhookConfig{}, err
}
contentType := hook.GetConfig().GetContentType()
if contentType == "" {
contentType = "json"
}
return WebhookConfig{
ID: hook.GetID(),
Events: hook.Events,
Active: hook.GetActive(),
URL: hook.GetConfig().GetURL(),
ContentType: contentType,
// Intentionally not setting Secret.
}, nil
}
func (r *githubClient) DeleteWebhook(ctx context.Context, owner, repository string, webhookID int64) error {
_, err := r.gh.Repositories.DeleteHook(ctx, owner, repository, webhookID)
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return err
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return ErrResourceNotFound
}
return err
}
func (r *githubClient) EditWebhook(ctx context.Context, owner, repository string, cfg WebhookConfig) error {
if cfg.ContentType == "" {
cfg.ContentType = "form"
}
hook := &github.Hook{
URL: &cfg.URL,
Events: cfg.Events,
Active: &cfg.Active,
Config: &github.HookConfig{
ContentType: &cfg.ContentType,
Secret: &cfg.Secret,
URL: &cfg.URL,
},
}
_, _, err := r.gh.Repositories.EditHook(ctx, owner, repository, cfg.ID, hook)
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) && ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
return err
}
func (r *githubClient) ListPullRequestFiles(ctx context.Context, owner, repository string, number int) ([]CommitFile, error) {
listFn := func(ctx context.Context, opts *github.ListOptions) ([]*github.CommitFile, *github.Response, error) {
return r.gh.PullRequests.ListFiles(ctx, owner, repository, number, opts)
}
files, err := paginatedList(
ctx,
listFn,
defaultListOptions(maxPRFiles),
)
if errors.Is(err, ErrTooManyItems) {
return nil, fmt.Errorf("pull request contains too many files (more than %d)", maxPRFiles)
}
if err != nil {
return nil, err
}
// Convert to the interface type
ret := make([]CommitFile, 0, len(files))
for _, f := range files {
ret = append(ret, f)
}
return ret, nil
}
func (r *githubClient) CreatePullRequestComment(ctx context.Context, owner, repository string, number int, body string) error {
comment := &github.IssueComment{
Body: &body,
}
if _, _, err := r.gh.Issues.CreateComment(ctx, owner, repository, number, comment); err != nil {
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) && ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
return err
}
return nil
}
func (r *githubClient) CreatePullRequestFileComment(ctx context.Context, owner, repository string, number int, comment FileComment) error {
commentRequest := &github.PullRequestComment{
Body: &comment.Content,
CommitID: &comment.Ref,
Path: &comment.Path,
Position: &comment.Position,
}
if _, _, err := r.gh.PullRequests.CreateComment(ctx, owner, repository, number, commentRequest); err != nil {
var ghErr *github.ErrorResponse
if errors.As(err, &ghErr) && ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return ErrServiceUnavailable
}
return err
}
return nil
}
func (r *githubClient) ClearAllPullRequestFileComments(ctx context.Context, owner, repository string, number int) error {
listFn := func(ctx context.Context, opts *github.ListOptions) ([]*github.PullRequestComment, *github.Response, error) {
return r.gh.PullRequests.ListComments(ctx, owner, repository, number, &github.PullRequestListCommentsOptions{
ListOptions: *opts,
})
}
comments, err := paginatedList(ctx, listFn, defaultListOptions(maxPullRequestsFileComments))
if errors.Is(err, ErrTooManyItems) {
return fmt.Errorf("too many comments to process (more than %d)", maxPullRequestsFileComments)
}
if err != nil {
return err
}
userLogin, _, err := r.gh.Users.Get(ctx, "")
if err != nil {
return fmt.Errorf("get user: %w", err)
}
for _, c := range comments {
// skip if comments were not created by us
if c.User.GetLogin() != userLogin.GetLogin() {
continue
}
if _, err := r.gh.PullRequests.DeleteComment(ctx, owner, repository, c.GetID()); err != nil {
return fmt.Errorf("delete comment: %w", err)
}
}
return nil
}
type realRepositoryContent struct {
real *github.RepositoryContent
}
var _ RepositoryContent = realRepositoryContent{}
func (c realRepositoryContent) IsDirectory() bool {
return c.real.GetType() == "dir"
}
func (c realRepositoryContent) GetFileContent() (string, error) {
return c.real.GetContent()
}
func (c realRepositoryContent) IsSymlink() bool {
return c.real.Target != nil
}
func (c realRepositoryContent) GetPath() string {
return c.real.GetPath()
}
func (c realRepositoryContent) GetSHA() string {
return c.real.GetSHA()
}
func (c realRepositoryContent) GetSize() int64 {
if c.real.Size != nil {
return int64(*c.real.Size)
}
if c.real.Content != nil {
if c, err := c.real.GetContent(); err == nil {
return int64(len(c))
}
}
return 0
}
// listOptions represents pagination parameters for list operations
type listOptions struct {
github.ListOptions
MaxItems int
}
// defaultListOptions returns a ListOptions with sensible defaults
func defaultListOptions(maxItems int) listOptions {
return listOptions{
ListOptions: github.ListOptions{
Page: 1,
PerPage: 100,
},
MaxItems: maxItems,
}
}
// paginatedList is a generic function to handle GitHub API pagination
func paginatedList[T any](
ctx context.Context,
listFn func(context.Context, *github.ListOptions) ([]T, *github.Response, error),
opts listOptions,
) ([]T, error) {
var allItems []T
for {
items, resp, err := listFn(ctx, &opts.ListOptions)
if err != nil {
var ghErr *github.ErrorResponse
if !errors.As(err, &ghErr) {
return nil, err
}
if ghErr.Response.StatusCode == http.StatusServiceUnavailable {
return nil, ErrServiceUnavailable
}
if ghErr.Response.StatusCode == http.StatusNotFound {
return nil, ErrResourceNotFound
}
return nil, err
}
// Pre-allocate the slice if this is the first page
if allItems == nil {
allItems = make([]T, 0, len(items)*2) // Estimate double the first page size
}
allItems = append(allItems, items...)
// Check if we've exceeded the maximum allowed items
if len(allItems) > opts.MaxItems {
return nil, ErrTooManyItems
}
// If there are no more pages, break
if resp.NextPage == 0 {
break
}
// Set up next page
opts.Page = resp.NextPage
}
return allItems, nil
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-issue_comment-created.json
Vendored
+231
View File
@@ -0,0 +1,231 @@
{
"action": "created",
"issue": {
"id": 2726065547,
"number": 12,
"state": "open",
"locked": false,
"title": "Webhook test PR",
"author_association": "MEMBER",
"user": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"comments": 1,
"created_at": "2024-12-09T05:53:14Z",
"updated_at": "2024-12-09T06:03:21Z",
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12",
"html_url": "https://github.com/grafana/git-ui-sync-demo/pull/12",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/comments",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/events",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/labels{/name}",
"repository_url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"pull_request": {
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12",
"html_url": "https://github.com/grafana/git-ui-sync-demo/pull/12",
"diff_url": "https://github.com/grafana/git-ui-sync-demo/pull/12.diff",
"patch_url": "https://github.com/grafana/git-ui-sync-demo/pull/12.patch"
},
"reactions": {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"confused": 0,
"heart": 0,
"hooray": 0,
"rocket": 0,
"eyes": 0,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/reactions"
},
"node_id": "PR_kwDONO4cS86EfDVR",
"draft": false
},
"comment": {
"id": 2527008082,
"node_id": "IC_kwDONO4cS86WnxVS",
"body": "comment in PR",
"user": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"reactions": {
"total_count": 0,
"+1": 0,
"-1": 0,
"laugh": 0,
"confused": 0,
"heart": 0,
"hooray": 0,
"rocket": 0,
"eyes": 0,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments/2527008082/reactions"
},
"created_at": "2024-12-09T06:03:19Z",
"updated_at": "2024-12-09T06:03:19Z",
"author_association": "MEMBER",
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments/2527008082",
"html_url": "https://github.com/grafana/git-ui-sync-demo/pull/12#issuecomment-2527008082",
"issue_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12"
},
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"default_branch": "main",
"created_at": "2024-11-13T17:13:33Z",
"pushed_at": "2024-12-09T05:58:00Z",
"updated_at": "2024-12-09T05:58:03Z",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"fork": false,
"forks_count": 0,
"open_issues_count": 9,
"open_issues": 9,
"stargazers_count": 0,
"watchers_count": 0,
"watchers": 0,
"size": 141,
"allow_forking": false,
"web_commit_signoff_required": false,
"archived": false,
"disabled": false,
"private": true,
"has_issues": true,
"has_wiki": true,
"has_pages": false,
"has_projects": true,
"has_downloads": true,
"has_discussions": false,
"is_template": false,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"visibility": "internal"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"organization": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open \u0026 composable observability platform.",
"url": "https://api.github.com/orgs/grafana",
"events_url": "https://api.github.com/orgs/grafana/events",
"hooks_url": "https://api.github.com/orgs/grafana/hooks",
"issues_url": "https://api.github.com/orgs/grafana/issues",
"members_url": "https://api.github.com/orgs/grafana/members{/member}",
"public_members_url": "https://api.github.com/orgs/grafana/public_members{/member}",
"repos_url": "https://api.github.com/orgs/grafana/repos"
}
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-ping-check.json
Vendored
+143
View File
@@ -0,0 +1,143 @@
{
"zen": "Keep it logically awesome.",
"hook_id": 517704995,
"hook": {
"created_at": "2024-12-09T05:44:20Z",
"updated_at": "2024-12-09T05:44:20Z",
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks/517704995",
"id": 517704995,
"type": "Repository",
"name": "web",
"test_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks/517704995/test",
"ping_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks/517704995/pings",
"last_response": {
"code": null,
"message": null,
"status": "unused"
},
"config": {
"content_type": "form",
"insecure_ssl": "0",
"url": "https://0b71-216-128-0-90.ngrok-free.app/apis/provisioning.grafana.app/v0alpha1/namespaces/default/repositories/github-example-ryan/webhook",
"secret": "********"
},
"events": [
"*"
],
"active": true
},
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"default_branch": "main",
"created_at": "2024-11-13T17:13:33Z",
"pushed_at": "2024-12-08T10:51:24Z",
"updated_at": "2024-11-28T12:53:26Z",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"fork": false,
"forks_count": 0,
"open_issues_count": 8,
"open_issues": 8,
"stargazers_count": 0,
"watchers_count": 0,
"watchers": 0,
"size": 141,
"allow_forking": false,
"web_commit_signoff_required": false,
"archived": false,
"disabled": false,
"private": true,
"has_issues": true,
"has_wiki": true,
"has_pages": false,
"has_projects": true,
"has_downloads": true,
"has_discussions": false,
"is_template": false,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"visibility": "internal"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
}
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-pull_request-opened.json
Vendored
+470
View File
@@ -0,0 +1,470 @@
{
"action": "opened",
"number": 12,
"pull_request": {
"id": 2222732625,
"number": 12,
"state": "open",
"locked": false,
"title": "Webhook test PR",
"created_at": "2024-12-09T05:53:14Z",
"updated_at": "2024-12-09T05:53:14Z",
"user": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"draft": false,
"merged": false,
"mergeable_state": "unknown",
"comments": 0,
"commits": 1,
"additions": 1,
"deletions": 0,
"changed_files": 1,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12",
"html_url": "https://github.com/grafana/git-ui-sync-demo/pull/12",
"issue_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/ab5446a53df9e5f8bdeed52250f51fad08e822bc",
"diff_url": "https://github.com/grafana/git-ui-sync-demo/pull/12.diff",
"patch_url": "https://github.com/grafana/git-ui-sync-demo/pull/12.patch",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12/commits",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/comments",
"review_comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12/comments",
"review_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/comments{/number}",
"review_comments": 0,
"maintainer_can_modify": false,
"author_association": "MEMBER",
"node_id": "PR_kwDONO4cS86EfDVR",
"_links": {
"self": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12"
},
"html": {
"href": "https://github.com/grafana/git-ui-sync-demo/pull/12"
},
"issue": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12"
},
"comments": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/12/comments"
},
"review_comments": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12/comments"
},
"review_comment": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/comments{/number}"
},
"commits": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls/12/commits"
},
"statuses": {
"href": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/ab5446a53df9e5f8bdeed52250f51fad08e822bc"
}
},
"head": {
"label": "grafana:dashboard/1733653266690",
"ref": "dashboard/1733653266690",
"sha": "ab5446a53df9e5f8bdeed52250f51fad08e822bc",
"repo": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"default_branch": "main",
"created_at": "2024-11-13T17:13:33Z",
"pushed_at": "2024-12-08T10:51:24Z",
"updated_at": "2024-11-28T12:53:26Z",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"fork": false,
"forks_count": 0,
"open_issues_count": 9,
"open_issues": 9,
"stargazers_count": 0,
"watchers_count": 0,
"watchers": 0,
"size": 141,
"allow_rebase_merge": true,
"allow_update_branch": false,
"allow_squash_merge": true,
"allow_merge_commit": true,
"allow_auto_merge": false,
"allow_forking": false,
"web_commit_signoff_required": false,
"delete_branch_on_merge": false,
"use_squash_pr_title_as_default": false,
"squash_merge_commit_title": "COMMIT_OR_PR_TITLE",
"squash_merge_commit_message": "COMMIT_MESSAGES",
"merge_commit_title": "MERGE_MESSAGE",
"merge_commit_message": "PR_TITLE",
"archived": false,
"disabled": false,
"private": true,
"has_issues": true,
"has_wiki": true,
"has_pages": false,
"has_projects": true,
"has_downloads": true,
"has_discussions": false,
"is_template": false,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"visibility": "internal"
},
"user": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
}
},
"base": {
"label": "grafana:main",
"ref": "main",
"sha": "6c86a0cdfd220c2fe3518cfaa4a4babf030d9a7a",
"repo": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"default_branch": "main",
"created_at": "2024-11-13T17:13:33Z",
"pushed_at": "2024-12-08T10:51:24Z",
"updated_at": "2024-11-28T12:53:26Z",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"fork": false,
"forks_count": 0,
"open_issues_count": 9,
"open_issues": 9,
"stargazers_count": 0,
"watchers_count": 0,
"watchers": 0,
"size": 141,
"allow_rebase_merge": true,
"allow_update_branch": false,
"allow_squash_merge": true,
"allow_merge_commit": true,
"allow_auto_merge": false,
"allow_forking": false,
"web_commit_signoff_required": false,
"delete_branch_on_merge": false,
"use_squash_pr_title_as_default": false,
"squash_merge_commit_title": "COMMIT_OR_PR_TITLE",
"squash_merge_commit_message": "COMMIT_MESSAGES",
"merge_commit_title": "MERGE_MESSAGE",
"merge_commit_message": "PR_TITLE",
"archived": false,
"disabled": false,
"private": true,
"has_issues": true,
"has_wiki": true,
"has_pages": false,
"has_projects": true,
"has_downloads": true,
"has_discussions": false,
"is_template": false,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"visibility": "internal"
},
"user": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
}
}
},
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"default_branch": "main",
"created_at": "2024-11-13T17:13:33Z",
"pushed_at": "2024-12-08T10:51:24Z",
"updated_at": "2024-11-28T12:53:26Z",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"fork": false,
"forks_count": 0,
"open_issues_count": 9,
"open_issues": 9,
"stargazers_count": 0,
"watchers_count": 0,
"watchers": 0,
"size": 141,
"allow_forking": false,
"web_commit_signoff_required": false,
"archived": false,
"disabled": false,
"private": true,
"has_issues": true,
"has_wiki": true,
"has_pages": false,
"has_projects": true,
"has_downloads": true,
"has_discussions": false,
"is_template": false,
"url": "https://api.github.com/repos/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"visibility": "internal"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"organization": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open \u0026 composable observability platform.",
"url": "https://api.github.com/orgs/grafana",
"events_url": "https://api.github.com/orgs/grafana/events",
"hooks_url": "https://api.github.com/orgs/grafana/hooks",
"issues_url": "https://api.github.com/orgs/grafana/issues",
"members_url": "https://api.github.com/orgs/grafana/members{/member}",
"public_members_url": "https://api.github.com/orgs/grafana/public_members{/member}",
"repos_url": "https://api.github.com/orgs/grafana/repos"
}
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-push-different_branch.json
Vendored
+148
View File
@@ -0,0 +1,148 @@
{
"ref": "refs/heads/not-main",
"commits": [
{
"message": "Update README.md\n\ntest message",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"url": "https://github.com/grafana/git-ui-sync-demo/commit/72096e3adc646c5a5b8a91744f962b12bac06045",
"distinct": true,
"id": "72096e3adc646c5a5b8a91744f962b12bac06045",
"tree_id": "03ff034c54bcefae2f96041f3fb8172f2fe93df3",
"timestamp": "2024-12-09T08:58:00+03:00",
"committer": {
"name": "GitHub",
"email": "noreply@github.com",
"username": "web-flow"
},
"modified": [
"README.md"
]
}
],
"before": "6c86a0cdfd220c2fe3518cfaa4a4babf030d9a7a",
"after": "72096e3adc646c5a5b8a91744f962b12bac06045",
"created": false,
"deleted": false,
"forced": false,
"compare": "https://github.com/grafana/git-ui-sync-demo/compare/6c86a0cdfd22...72096e3adc64",
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"name": "grafana",
"email": "hello@grafana.com",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"private": true,
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"fork": false,
"created_at": "2024-11-13T20:13:33+03:00",
"pushed_at": "2024-12-09T08:58:00+03:00",
"updated_at": "2024-11-28T12:53:26Z",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"size": 141,
"stargazers_count": 0,
"watchers_count": 0,
"has_issues": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"archived": false,
"disabled": false,
"open_issues_count": 9,
"default_branch": "main",
"master_branch": "main",
"organization": "grafana",
"url": "https://github.com/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo"
},
"head_commit": {
"message": "Update README.md\n\ntest message",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"url": "https://github.com/grafana/git-ui-sync-demo/commit/72096e3adc646c5a5b8a91744f962b12bac06045",
"distinct": true,
"id": "72096e3adc646c5a5b8a91744f962b12bac06045",
"tree_id": "03ff034c54bcefae2f96041f3fb8172f2fe93df3",
"timestamp": "2024-12-09T08:58:00+03:00",
"committer": {
"name": "GitHub",
"email": "noreply@github.com",
"username": "web-flow"
},
"modified": [
"README.md"
]
},
"pusher": {
"name": "ryantxu",
"email": "ryantxu@gmail.com"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"organization": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open \u0026 composable observability platform.",
"url": "https://api.github.com/orgs/grafana",
"events_url": "https://api.github.com/orgs/grafana/events",
"hooks_url": "https://api.github.com/orgs/grafana/hooks",
"issues_url": "https://api.github.com/orgs/grafana/issues",
"members_url": "https://api.github.com/orgs/grafana/members{/member}",
"public_members_url": "https://api.github.com/orgs/grafana/public_members{/member}",
"repos_url": "https://api.github.com/orgs/grafana/repos"
}
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-push-nested.json
Vendored
+234
View File
@@ -0,0 +1,234 @@
{
"ref": "refs/heads/main",
"before": "72096e3adc646c5a5b8a91744f962b12bac06045",
"after": "5c816f9812e391c62b0c5555d0b473b296d9179c",
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"private": true,
"owner": {
"name": "grafana",
"email": "hello@grafana.com",
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/grafana",
"html_url": "https://github.com/grafana",
"followers_url": "https://api.github.com/users/grafana/followers",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"repos_url": "https://api.github.com/users/grafana/repos",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"type": "Organization",
"user_view_type": "public",
"site_admin": false
},
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"fork": false,
"url": "https://github.com/grafana/git-ui-sync-demo",
"forks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/forks",
"keys_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/keys{/key_id}",
"collaborators_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/collaborators{/collaborator}",
"teams_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/teams",
"hooks_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/hooks",
"issue_events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/events{/number}",
"events_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/events",
"assignees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/assignees{/user}",
"branches_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/branches{/branch}",
"tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/tags",
"blobs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/blobs{/sha}",
"git_tags_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/tags{/sha}",
"git_refs_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/refs{/sha}",
"trees_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/trees{/sha}",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"languages_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/languages",
"stargazers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/stargazers",
"contributors_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contributors",
"subscribers_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscribers",
"subscription_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/subscription",
"commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/commits{/sha}",
"git_commits_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/git/commits{/sha}",
"comments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/comments{/number}",
"issue_comment_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues/comments{/number}",
"contents_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/contents/{+path}",
"compare_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/compare/{base}...{head}",
"merges_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/merges",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"downloads_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/downloads",
"issues_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/issues{/number}",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"milestones_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/milestones{/number}",
"notifications_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/notifications{?since,all,participating}",
"labels_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/labels{/name}",
"releases_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/releases{/id}",
"deployments_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/deployments",
"created_at": 1731518013,
"updated_at": "2024-12-09T05:58:03Z",
"pushed_at": 1733731254,
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo",
"homepage": null,
"size": 142,
"stargazers_count": 0,
"watchers_count": 0,
"language": null,
"has_issues": true,
"has_projects": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"has_discussions": false,
"forks_count": 0,
"mirror_url": null,
"archived": false,
"disabled": false,
"open_issues_count": 9,
"license": null,
"allow_forking": false,
"is_template": false,
"web_commit_signoff_required": false,
"topics": [
],
"visibility": "internal",
"forks": 0,
"open_issues": 9,
"watchers": 0,
"default_branch": "main",
"stargazers": 0,
"master_branch": "main",
"organization": "grafana",
"custom_properties": {
}
},
"pusher": {
"name": "ryantxu",
"email": "ryantxu@gmail.com"
},
"organization": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"url": "https://api.github.com/orgs/grafana",
"repos_url": "https://api.github.com/orgs/grafana/repos",
"events_url": "https://api.github.com/orgs/grafana/events",
"hooks_url": "https://api.github.com/orgs/grafana/hooks",
"issues_url": "https://api.github.com/orgs/grafana/issues",
"members_url": "https://api.github.com/orgs/grafana/members{/member}",
"public_members_url": "https://api.github.com/orgs/grafana/public_members{/member}",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open & composable observability platform."
},
"enterprise": {
"id": 129980,
"slug": "grafana",
"name": "Grafana Labs Enterprise",
"node_id": "E_kgDOAAH7vA",
"avatar_url": "https://avatars.githubusercontent.com/b/129980?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open & composable observability platform.",
"website_url": "https://grafana.com",
"html_url": "https://github.com/enterprises/grafana",
"created_at": "2024-02-29T23:01:47Z",
"updated_at": "2024-10-21T08:45:28Z"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"gravatar_id": "",
"url": "https://api.github.com/users/ryantxu",
"html_url": "https://github.com/ryantxu",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"type": "User",
"user_view_type": "public",
"site_admin": false
},
"created": false,
"deleted": false,
"forced": false,
"base_ref": null,
"compare": "https://github.com/grafana/git-ui-sync-demo/compare/72096e3adc64...5c816f9812e3",
"commits": [
{
"id": "5c816f9812e391c62b0c5555d0b473b296d9179c",
"tree_id": "97c7ba756b07b6a2b7fd130e59f75deed53fe027",
"distinct": true,
"message": "nested folders",
"timestamp": "2024-12-09T11:00:48+03:00",
"url": "https://github.com/grafana/git-ui-sync-demo/commit/5c816f9812e391c62b0c5555d0b473b296d9179c",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"committer": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"added": [
"nested-1/README.md",
"nested-1/dash-1.json",
"nested-1/nested-2/README.md",
"nested-1/nested-2/dash-2.json"
],
"removed": [
],
"modified": [
"first-dashboard.json"
]
}
],
"head_commit": {
"id": "5c816f9812e391c62b0c5555d0b473b296d9179c",
"tree_id": "97c7ba756b07b6a2b7fd130e59f75deed53fe027",
"distinct": true,
"message": "nested folders",
"timestamp": "2024-12-09T11:00:48+03:00",
"url": "https://github.com/grafana/git-ui-sync-demo/commit/5c816f9812e391c62b0c5555d0b473b296d9179c",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"committer": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"added": [
"nested-1/README.md",
"nested-1/dash-1.json",
"nested-1/nested-2/README.md",
"nested-1/nested-2/dash-2.json"
],
"removed": [
],
"modified": [
"first-dashboard.json"
]
}
}
pkg/registry/apis/provisioning/repository/github/testdata/webhook-push-nothing_relevant.json
Vendored
+148
View File
@@ -0,0 +1,148 @@
{
"ref": "refs/heads/main",
"commits": [
{
"message": "Update README.md\n\ntest message",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"url": "https://github.com/grafana/git-ui-sync-demo/commit/72096e3adc646c5a5b8a91744f962b12bac06045",
"distinct": true,
"id": "72096e3adc646c5a5b8a91744f962b12bac06045",
"tree_id": "03ff034c54bcefae2f96041f3fb8172f2fe93df3",
"timestamp": "2024-12-09T08:58:00+03:00",
"committer": {
"name": "GitHub",
"email": "noreply@github.com",
"username": "web-flow"
},
"modified": [
"README.md"
]
}
],
"before": "6c86a0cdfd220c2fe3518cfaa4a4babf030d9a7a",
"after": "72096e3adc646c5a5b8a91744f962b12bac06045",
"created": false,
"deleted": false,
"forced": false,
"compare": "https://github.com/grafana/git-ui-sync-demo/compare/6c86a0cdfd22...72096e3adc64",
"repository": {
"id": 888020043,
"node_id": "R_kgDONO4cSw",
"name": "git-ui-sync-demo",
"full_name": "grafana/git-ui-sync-demo",
"owner": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"html_url": "https://github.com/grafana",
"gravatar_id": "",
"name": "grafana",
"email": "hello@grafana.com",
"type": "Organization",
"site_admin": false,
"url": "https://api.github.com/users/grafana",
"events_url": "https://api.github.com/users/grafana/events{/privacy}",
"following_url": "https://api.github.com/users/grafana/following{/other_user}",
"followers_url": "https://api.github.com/users/grafana/followers",
"gists_url": "https://api.github.com/users/grafana/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/grafana/orgs",
"received_events_url": "https://api.github.com/users/grafana/received_events",
"repos_url": "https://api.github.com/users/grafana/repos",
"starred_url": "https://api.github.com/users/grafana/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/grafana/subscriptions"
},
"private": true,
"description": "A repository containing Grafana dashboards to demo the Github Sync feature in Grafana.",
"fork": false,
"created_at": "2024-11-13T20:13:33+03:00",
"pushed_at": "2024-12-09T08:58:00+03:00",
"updated_at": "2024-11-28T12:53:26Z",
"pulls_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/pulls{/number}",
"size": 141,
"stargazers_count": 0,
"watchers_count": 0,
"has_issues": true,
"has_downloads": true,
"has_wiki": true,
"has_pages": false,
"forks_count": 0,
"archived": false,
"disabled": false,
"open_issues_count": 9,
"default_branch": "main",
"master_branch": "main",
"organization": "grafana",
"url": "https://github.com/grafana/git-ui-sync-demo",
"archive_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/{archive_format}{/ref}",
"html_url": "https://github.com/grafana/git-ui-sync-demo",
"statuses_url": "https://api.github.com/repos/grafana/git-ui-sync-demo/statuses/{sha}",
"git_url": "git://github.com/grafana/git-ui-sync-demo.git",
"ssh_url": "git@github.com:grafana/git-ui-sync-demo.git",
"clone_url": "https://github.com/grafana/git-ui-sync-demo.git",
"svn_url": "https://github.com/grafana/git-ui-sync-demo"
},
"head_commit": {
"message": "Update README.md\n\ntest message",
"author": {
"name": "Ryan McKinley",
"email": "ryantxu@gmail.com",
"username": "ryantxu"
},
"url": "https://github.com/grafana/git-ui-sync-demo/commit/72096e3adc646c5a5b8a91744f962b12bac06045",
"distinct": true,
"id": "72096e3adc646c5a5b8a91744f962b12bac06045",
"tree_id": "03ff034c54bcefae2f96041f3fb8172f2fe93df3",
"timestamp": "2024-12-09T08:58:00+03:00",
"committer": {
"name": "GitHub",
"email": "noreply@github.com",
"username": "web-flow"
},
"modified": [
"README.md"
]
},
"pusher": {
"name": "ryantxu",
"email": "ryantxu@gmail.com"
},
"sender": {
"login": "ryantxu",
"id": 705951,
"node_id": "MDQ6VXNlcjcwNTk1MQ==",
"avatar_url": "https://avatars.githubusercontent.com/u/705951?v=4",
"html_url": "https://github.com/ryantxu",
"gravatar_id": "",
"type": "User",
"site_admin": false,
"url": "https://api.github.com/users/ryantxu",
"events_url": "https://api.github.com/users/ryantxu/events{/privacy}",
"following_url": "https://api.github.com/users/ryantxu/following{/other_user}",
"followers_url": "https://api.github.com/users/ryantxu/followers",
"gists_url": "https://api.github.com/users/ryantxu/gists{/gist_id}",
"organizations_url": "https://api.github.com/users/ryantxu/orgs",
"received_events_url": "https://api.github.com/users/ryantxu/received_events",
"repos_url": "https://api.github.com/users/ryantxu/repos",
"starred_url": "https://api.github.com/users/ryantxu/starred{/owner}{/repo}",
"subscriptions_url": "https://api.github.com/users/ryantxu/subscriptions"
},
"organization": {
"login": "grafana",
"id": 7195757,
"node_id": "MDEyOk9yZ2FuaXphdGlvbjcxOTU3NTc=",
"avatar_url": "https://avatars.githubusercontent.com/u/7195757?v=4",
"description": "Grafana Labs is behind leading open source projects Grafana and Loki, and the creator of the first open \u0026 composable observability platform.",
"url": "https://api.github.com/orgs/grafana",
"events_url": "https://api.github.com/orgs/grafana/events",
"hooks_url": "https://api.github.com/orgs/grafana/hooks",
"issues_url": "https://api.github.com/orgs/grafana/issues",
"members_url": "https://api.github.com/orgs/grafana/members{/member}",
"public_members_url": "https://api.github.com/orgs/grafana/public_members{/member}",
"repos_url": "https://api.github.com/orgs/grafana/repos"
}
}
pkg/registry/apis/provisioning/repository/github_test.go
+149
View File
@@ -0,0 +1,149 @@
package repository
import (
"fmt"
"net/http"
"os"
"path"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
)
func TestIsValidGitBranchName(t *testing.T) {
tests := []struct {
name string
branch string
expected bool
}{
{"Valid branch name", "feature/add-tests", true},
{"Valid branch name with numbers", "feature/123-add-tests", true},
{"Valid branch name with dots", "feature.add.tests", true},
{"Valid branch name with hyphens", "feature-add-tests", true},
{"Valid branch name with underscores", "feature_add_tests", true},
{"Valid branch name with mixed characters", "feature/add_tests-123", true},
{"Starts with /", "/feature", false},
{"Ends with /", "feature/", false},
{"Ends with .", "feature.", false},
{"Ends with space", "feature ", false},
{"Contains consecutive slashes", "feature//branch", false},
{"Contains consecutive dots", "feature..branch", false},
{"Contains @{", "feature@{branch", false},
{"Contains invalid character ~", "feature~branch", false},
{"Contains invalid character ^", "feature^branch", false},
{"Contains invalid character :", "feature:branch", false},
{"Contains invalid character ?", "feature?branch", false},
{"Contains invalid character *", "feature*branch", false},
{"Contains invalid character [", "feature[branch", false},
{"Contains invalid character ]", "feature]branch", false},
{"Contains invalid character \\", "feature\\branch", false},
{"Empty branch name", "", false},
{"Only whitespace", " ", false},
{"Single valid character", "a", true},
{"Ends with .lock", "feature.lock", false},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
assert.Equal(t, tt.expected, isValidGitBranchName(tt.branch))
})
}
}
func TestParseWebhooks(t *testing.T) {
tests := []struct {
messageType string
name string
expected provisioning.WebhookResponse
}{
{"ping", "check", provisioning.WebhookResponse{
Code: http.StatusOK,
}},
{"pull_request", "opened", provisioning.WebhookResponse{
Code: http.StatusAccepted, // 202
Job: &provisioning.JobSpec{
Repository: "unit-test-repo",
Action: provisioning.JobActionPullRequest,
PullRequest: &provisioning.PullRequestJobOptions{
Ref: "dashboard/1733653266690",
Hash: "ab5446a53df9e5f8bdeed52250f51fad08e822bc",
PR: 12,
URL: "https://github.com/grafana/git-ui-sync-demo/pull/12",
},
},
}},
{"push", "different_branch", provisioning.WebhookResponse{
Code: http.StatusOK, // we don't care about a branch that isn't the one we configured
}},
{"push", "nothing_relevant", provisioning.WebhookResponse{
Code: http.StatusAccepted,
Job: &provisioning.JobSpec{ // we want to always push a sync job
Repository: "unit-test-repo",
Action: provisioning.JobActionSync,
Pull: &provisioning.SyncJobOptions{
Incremental: true,
},
},
}},
{"push", "nested", provisioning.WebhookResponse{
Code: http.StatusAccepted,
Job: &provisioning.JobSpec{
Repository: "unit-test-repo",
Action: provisioning.JobActionSync,
Pull: &provisioning.SyncJobOptions{
Incremental: true,
},
},
}},
{"issue_comment", "created", provisioning.WebhookResponse{
Code: http.StatusNotImplemented,
}},
}
gh := &githubRepository{
config: &provisioning.Repository{
ObjectMeta: v1.ObjectMeta{
Name: "unit-test-repo",
},
Spec: provisioning.RepositorySpec{
Sync: provisioning.SyncOptions{
Enabled: true, // required to accept sync job
},
GitHub: &provisioning.GitHubRepositoryConfig{
URL: "https://github.com/grafana/git-ui-sync-demo",
Branch: "main",
GenerateDashboardPreviews: true,
},
},
},
}
var err error
gh.owner, gh.repo, err = parseOwnerRepo(gh.config.Spec.GitHub.URL)
require.NoError(t, err)
// Support parsing from a ".git" extension
owner, repo, err := parseOwnerRepo(gh.config.Spec.GitHub.URL + ".git")
require.NoError(t, err)
require.Equal(t, gh.owner, owner)
require.Equal(t, gh.repo, repo)
for _, tt := range tests {
name := fmt.Sprintf("webhook-%s-%s.json", tt.messageType, tt.name)
t.Run(name, func(t *testing.T) {
// nolint:gosec
payload, err := os.ReadFile(path.Join("github", "testdata", name))
require.NoError(t, err)
rsp, err := gh.parseWebhook(tt.messageType, payload)
require.NoError(t, err)
require.Equal(t, tt.expected.Code, rsp.Code)
require.Equal(t, tt.expected.Job, rsp.Job)
})
}
}
pkg/registry/apis/provisioning/repository/go-git/transport.go
+71
View File
@@ -0,0 +1,71 @@
package gogit
import (
"fmt"
"io"
"net/http"
"sync/atomic"
)
var errBytesLimitExceeded = fmt.Errorf("bytes limit exceeded")
// ByteLimitedTransport wraps http.RoundTripper to enforce a max byte limit
type ByteLimitedTransport struct {
Transport http.RoundTripper
Limit int64
Bytes int64
}
// NewByteLimitedTransport creates a new ByteLimitedTransport with the specified transport and byte limit.
// If transport is nil, http.DefaultTransport will be used.
func NewByteLimitedTransport(transport http.RoundTripper, limit int64) *ByteLimitedTransport {
if transport == nil {
transport = http.DefaultTransport
}
return &ByteLimitedTransport{
Transport: transport,
Limit: limit,
Bytes: 0,
}
}
// RoundTrip tracks downloaded bytes and aborts if limit is exceeded
func (b *ByteLimitedTransport) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := b.Transport.RoundTrip(req)
if err != nil {
return nil, err
}
// Wrap response body to track bytes read
resp.Body = &byteLimitedReader{
reader: resp.Body,
limit: b.Limit,
bytes: &b.Bytes,
}
return resp, nil
}
// byteLimitedReader tracks and enforces a download limit
type byteLimitedReader struct {
reader io.ReadCloser
limit int64
bytes *int64
}
func (r *byteLimitedReader) Read(p []byte) (int, error) {
n, err := r.reader.Read(p)
if err != nil {
return n, err
}
if atomic.AddInt64(r.bytes, int64(n)) > r.limit {
return 0, errBytesLimitExceeded
}
return n, nil
}
func (r *byteLimitedReader) Close() error {
return r.reader.Close()
}
pkg/registry/apis/provisioning/repository/go-git/transport_test.go
+139
View File
@@ -0,0 +1,139 @@
package gogit
import (
"bytes"
"errors"
"io"
"net/http"
"sync/atomic"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
type mockTransport struct {
response *http.Response
err error
}
func (m *mockTransport) RoundTrip(*http.Request) (*http.Response, error) {
return m.response, m.err
}
func TestNewByteLimitedTransport(t *testing.T) {
tests := []struct {
name string
transport http.RoundTripper
limit int64
}{
{
name: "with custom transport",
transport: &mockTransport{},
limit: 1000,
},
{
name: "with nil transport",
transport: nil,
limit: 1000,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
blt := NewByteLimitedTransport(tt.transport, tt.limit)
assert.NotNil(t, blt)
assert.Equal(t, tt.limit, blt.Limit)
assert.Equal(t, int64(0), blt.Bytes)
if tt.transport == nil {
assert.Equal(t, http.DefaultTransport, blt.Transport)
} else {
assert.Equal(t, tt.transport, blt.Transport)
}
})
}
}
func TestByteLimitedTransport_RoundTrip(t *testing.T) {
tests := []struct {
name string
responseBody string
limit int64
expectedError error
}{
{
name: "under limit",
responseBody: "small response",
limit: 100,
expectedError: nil,
},
{
name: "exceeds limit",
responseBody: "this response will exceed the byte limit",
limit: 10,
expectedError: errBytesLimitExceeded,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
mockResp := &http.Response{
Body: io.NopCloser(bytes.NewBufferString(tt.responseBody)),
}
mockTransport := &mockTransport{response: mockResp}
blt := NewByteLimitedTransport(mockTransport, tt.limit)
resp, err := blt.RoundTrip(&http.Request{})
require.NoError(t, err)
defer func() {
closeErr := resp.Body.Close()
assert.NoError(t, closeErr, "failed to close response body")
}()
data, err := io.ReadAll(resp.Body)
if tt.expectedError != nil {
assert.True(t, errors.Is(err, tt.expectedError), "expected error %v, got %v", tt.expectedError, err)
} else {
assert.NoError(t, err)
assert.Equal(t, tt.responseBody, string(data))
}
})
}
}
func TestByteLimitedReader_Close(t *testing.T) {
mockBody := io.NopCloser(bytes.NewBufferString("test"))
var byteCount int64
reader := &byteLimitedReader{
reader: mockBody,
limit: 100,
bytes: &byteCount,
}
err := reader.Close()
assert.NoError(t, err)
}
func TestByteLimitedReader_AtomicCounting(t *testing.T) {
var byteCount int64
reader := &byteLimitedReader{
reader: io.NopCloser(bytes.NewBufferString("test data")),
limit: 5,
bytes: &byteCount,
}
// First read should succeed
buf := make([]byte, 4)
n, err := reader.Read(buf)
assert.NoError(t, err)
assert.Equal(t, 4, n)
// Second read should fail due to limit
n, err = reader.Read(buf)
assert.True(t, errors.Is(err, errBytesLimitExceeded), "expected error %v, got %v", errBytesLimitExceeded, err)
assert.Equal(t, 0, n)
// Verify atomic counter
assert.Greater(t, atomic.LoadInt64(&byteCount), int64(5))
}
pkg/registry/apis/provisioning/repository/go-git/wrapper.go
+423
View File
@@ -0,0 +1,423 @@
package gogit
import (
"context"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"strings"
"time"
"github.com/go-git/go-billy/v5/util"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/go-git/go-git/v5/plumbing/transport/client"
githttp "github.com/go-git/go-git/v5/plumbing/transport/http"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/secrets"
)
const (
// maxOperationBytes is the maximum size of a git operation in bytes (1 GB)
maxOperationBytes = int64(1 << 30)
maxOperationTimeout = 10 * time.Minute
)
func init() {
// Create a size-limited writer that will cancel the context if size is exceeded
limitedTransport := NewByteLimitedTransport(http.DefaultTransport, maxOperationBytes)
httpClient := githttp.NewClient(&http.Client{
Transport: limitedTransport,
})
client.InstallProtocol("https", httpClient)
client.InstallProtocol("http", httpClient)
}
var _ repository.Repository = (*GoGitRepo)(nil)
type GoGitCloneOptions struct {
Root string // tempdir (when empty, memory??)
// If the branch does not exist, create it
CreateIfNotExists bool
// Skip intermediate commits and commit all before push
SingleCommitBeforePush bool
// Maximum allowed size for repository clone in bytes (0 means no limit)
MaxSize int64
// Maximum time allowed for clone operation in seconds (0 means no limit)
Timeout time.Duration
}
type GoGitPushOptions struct {
Timeout time.Duration
}
type GoGitRepo struct {
config *provisioning.Repository
opts GoGitCloneOptions
decryptedPassword string
repo *git.Repository
tree *git.Worktree
dir string // file path to worktree root (necessary? should use billy)
}
// This will create a new clone every time
// As structured, it is valid for one context and should not be shared across multiple requests
func Clone(
ctx context.Context,
config *provisioning.Repository,
opts GoGitCloneOptions,
secrets secrets.Service,
progress io.Writer, // os.Stdout
) (*GoGitRepo, error) {
if opts.Root == "" {
return nil, fmt.Errorf("missing root config")
}
// add a timeout to the operation
timeout := maxOperationTimeout
if opts.Timeout > 0 {
timeout = opts.Timeout
}
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
decrypted, err := secrets.Decrypt(ctx, config.Spec.GitHub.EncryptedToken)
if err != nil {
return nil, fmt.Errorf("error decrypting token: %w", err)
}
if err := os.MkdirAll(opts.Root, 0700); err != nil {
return nil, fmt.Errorf("create root dir: %w", err)
}
dir, err := mkdirTempClone(opts.Root, config)
if err != nil {
return nil, fmt.Errorf("create temp clone dir: %w", err)
}
repo, worktree, err := clone(ctx, config, opts, decrypted, dir, progress)
if err != nil {
if err := os.RemoveAll(dir); err != nil {
return nil, fmt.Errorf("remove temp clone dir after clone failed: %w", err)
}
return nil, fmt.Errorf("clone: %w", err)
}
return &GoGitRepo{
config: config,
opts: opts,
tree: worktree,
decryptedPassword: string(decrypted),
repo: repo,
dir: dir,
}, nil
}
func clone(ctx context.Context, config *provisioning.Repository, opts GoGitCloneOptions, decrypted []byte, dir string, progress io.Writer) (*git.Repository, *git.Worktree, error) {
gitcfg := config.Spec.GitHub
url := fmt.Sprintf("%s.git", gitcfg.URL)
branch := plumbing.NewBranchReferenceName(gitcfg.Branch)
cloneOpts := &git.CloneOptions{
ReferenceName: branch,
Auth: &githttp.BasicAuth{
Username: "grafana", // this can be anything except an empty string for PAT
Password: string(decrypted), // TODO... will need to get from a service!
},
URL: url,
Progress: progress,
}
repo, err := git.PlainCloneContext(ctx, dir, false, cloneOpts)
if errors.Is(err, plumbing.ErrReferenceNotFound) && opts.CreateIfNotExists {
cloneOpts.ReferenceName = "" // empty
repo, err = git.PlainCloneContext(ctx, dir, false, cloneOpts)
if err == nil {
worktree, err := repo.Worktree()
if err != nil {
return nil, nil, err
}
err = worktree.Checkout(&git.CheckoutOptions{
Branch: branch,
Force: true,
Create: true,
})
if err != nil {
return nil, nil, fmt.Errorf("unable to create new branch: %w", err)
}
}
} else if err != nil {
return nil, nil, fmt.Errorf("clone error: %w", err)
}
rcfg, err := repo.Config()
if err != nil {
return nil, nil, fmt.Errorf("error reading repository config %w", err)
}
origin := rcfg.Remotes["origin"]
if origin == nil {
return nil, nil, fmt.Errorf("missing origin remote %w", err)
}
if url != origin.URLs[0] {
return nil, nil, fmt.Errorf("unexpected remote (expected: %s, found: %s)", url, origin.URLs[0])
}
worktree, err := repo.Worktree()
if err != nil {
return nil, nil, fmt.Errorf("get worktree: %w", err)
}
return repo, worktree, nil
}
func mkdirTempClone(root string, config *provisioning.Repository) (string, error) {
if config.Namespace == "" {
return "", fmt.Errorf("config is missing namespace")
}
if config.Name == "" {
return "", fmt.Errorf("config is missing name")
}
return os.MkdirTemp(root, fmt.Sprintf("clone-%s-%s-", config.Namespace, config.Name))
}
// Affer making changes to the worktree, push changes
func (g *GoGitRepo) Push(ctx context.Context, opts GoGitPushOptions, progress io.Writer) error {
timeout := maxOperationTimeout
if opts.Timeout > 0 {
timeout = opts.Timeout
}
ctx, cancel := context.WithTimeout(ctx, timeout)
defer cancel()
if g.opts.SingleCommitBeforePush {
_, err := g.tree.Commit("exported from grafana", &git.CommitOptions{
All: true, // Add everything that changed
})
if err != nil {
// empty commit is fine -- no change
if !errors.Is(err, git.ErrEmptyCommit) {
return err
}
}
}
err := g.repo.PushContext(ctx, &git.PushOptions{
Progress: progress,
Force: true, // avoid fast-forward-errors
Auth: &githttp.BasicAuth{ // reuse logic from clone?
Username: "grafana",
Password: g.decryptedPassword,
},
})
if errors.Is(err, git.NoErrAlreadyUpToDate) {
return nil // same as the target
}
return err
}
func (g *GoGitRepo) Remove(ctx context.Context) error {
return os.RemoveAll(g.dir)
}
// Config implements repository.Repository.
func (g *GoGitRepo) Config() *provisioning.Repository {
return g.config
}
// ReadTree implements repository.Repository.
func (g *GoGitRepo) ReadTree(ctx context.Context, ref string) ([]repository.FileTreeEntry, error) {
var treePath string
if g.config.Spec.GitHub.Path != "" {
treePath = g.config.Spec.GitHub.Path
}
// TODO: do we really need this?
if !strings.HasPrefix(treePath, "/") {
treePath = "/" + treePath
}
entries := make([]repository.FileTreeEntry, 0, 100)
err := util.Walk(g.tree.Filesystem, treePath, func(path string, info fs.FileInfo, err error) error {
if err != nil || path == "/" {
return err
}
entry := repository.FileTreeEntry{
Path: strings.TrimLeft(path, "/"),
Size: info.Size(),
}
if !info.IsDir() {
entry.Blob = true
// For a real instance, this will likely be based on:
// https://github.com/go-git/go-git/blob/main/_examples/ls/main.go#L25
entry.Hash = fmt.Sprintf("TODO/%d", info.Size()) // but not used for
}
entries = append(entries, entry)
return err
})
if errors.Is(err, fs.ErrNotExist) {
// We intentionally ignore this case, as
} else if err != nil {
return nil, fmt.Errorf("failed to walk tree for ref '%s': %w", ref, err)
}
return entries, nil
}
func (g *GoGitRepo) Test(ctx context.Context) (*provisioning.TestResults, error) {
return &provisioning.TestResults{
Success: g.tree != nil,
}, nil
}
// Update implements repository.Repository.
func (g *GoGitRepo) Update(ctx context.Context, path string, ref string, data []byte, message string) error {
return g.Write(ctx, path, ref, data, message)
}
// Create implements repository.Repository.
func (g *GoGitRepo) Create(ctx context.Context, path string, ref string, data []byte, message string) error {
return g.Write(ctx, path, ref, data, message)
}
// Write implements repository.Repository.
func (g *GoGitRepo) Write(ctx context.Context, fpath string, ref string, data []byte, message string) error {
fpath = safepath.Join(g.config.Spec.GitHub.Path, fpath)
if err := verifyPathWithoutRef(fpath, ref); err != nil {
return err
}
// For folders, just create the folder and ignore the commit
if safepath.IsDir(fpath) {
return g.tree.Filesystem.MkdirAll(fpath, 0750)
}
dir := safepath.Dir(fpath)
if dir != "" {
err := g.tree.Filesystem.MkdirAll(dir, 0750)
if err != nil {
return err
}
}
file, err := g.tree.Filesystem.Create(fpath)
if err != nil {
return err
}
_, err = file.Write(data)
if err != nil {
return err
}
_, err = g.tree.Add(fpath)
if err != nil {
return err
}
// Skip commit for each file
if g.opts.SingleCommitBeforePush {
return nil
}
opts := &git.CommitOptions{}
sig := repository.GetAuthorSignature(ctx)
if sig != nil {
opts.Author = &object.Signature{
Name: sig.Name,
Email: sig.Email,
When: sig.When,
}
}
_, err = g.tree.Commit(message, opts)
if errors.Is(err, git.ErrEmptyCommit) {
return nil // empty commit is fine -- no change
}
return err
}
// Delete implements repository.Repository.
func (g *GoGitRepo) Delete(ctx context.Context, path string, ref string, message string) error {
if _, err := g.tree.Remove(safepath.Join(g.config.Spec.GitHub.Path, path)); err != nil {
return err
}
return nil
}
// Read implements repository.Repository.
func (g *GoGitRepo) Read(ctx context.Context, path string, ref string) (*repository.FileInfo, error) {
readPath := safepath.Join(g.config.Spec.GitHub.Path, path)
stat, err := g.tree.Filesystem.Lstat(readPath)
if err != nil {
return nil, fmt.Errorf("failed to stat path '%s': %w", readPath, err)
}
info := &repository.FileInfo{
Path: path,
Modified: &metav1.Time{
Time: stat.ModTime(),
},
}
if !stat.IsDir() {
f, err := g.tree.Filesystem.Open(readPath)
if err != nil {
return nil, err
}
info.Data, err = io.ReadAll(f)
if err != nil {
return nil, err
}
}
return info, err
}
func verifyPathWithoutRef(path string, ref string) error {
if path == "" {
return fmt.Errorf("expected path")
}
if ref != "" {
return fmt.Errorf("ref unsupported")
}
return nil
}
// History implements repository.Repository.
func (g *GoGitRepo) History(ctx context.Context, path string, ref string) ([]provisioning.HistoryItem, error) {
return nil, &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "history is not yet implemented",
Code: http.StatusNotImplemented,
},
}
}
// Validate implements repository.Repository.
func (g *GoGitRepo) Validate() field.ErrorList {
return nil
}
// Webhook implements repository.Repository.
func (g *GoGitRepo) Webhook(ctx context.Context, req *http.Request) (*provisioning.WebhookResponse, error) {
return nil, &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "history is not yet implemented",
Code: http.StatusNotImplemented,
},
}
}
pkg/registry/apis/provisioning/repository/go-git/wrapper_test.go
+94
View File
@@ -0,0 +1,94 @@
package gogit
import (
"context"
"encoding/json"
"fmt"
"os"
"testing"
"time"
"github.com/stretchr/testify/require"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
type dummySecret struct{}
// Decrypt implements secrets.Service.
func (d *dummySecret) Decrypt(ctx context.Context, data []byte) ([]byte, error) {
token, ok := os.LookupEnv("gitwraptoken")
if !ok {
return nil, fmt.Errorf("missing token in environment")
}
return []byte(token), nil
}
// Encrypt implements secrets.Service.
func (d *dummySecret) Encrypt(ctx context.Context, data []byte) ([]byte, error) {
panic("unimplemented")
}
// FIXME!! NOTE!!!!!
// This is really just a sketchpad while trying to get things working
// the test makes destructive changes to a real git repository :)
// this should be removed before committing to main (likely sooner)
// and replaced with integration tests that check the more specific results
func TestGoGitWrapper(t *testing.T) {
_, ok := os.LookupEnv("gitwraptoken")
if !ok {
t.Skipf("no token found in environment")
}
ctx := context.Background()
wrap, err := Clone(ctx, &v0alpha1.Repository{
ObjectMeta: v1.ObjectMeta{
Namespace: "ns",
Name: "unit-tester",
},
Spec: v0alpha1.RepositorySpec{
GitHub: &v0alpha1.GitHubRepositoryConfig{
URL: "https://github.com/grafana/git-ui-sync-demo",
Branch: "ryan-test",
},
},
},
GoGitCloneOptions{
Root: "testdata/clone", // where things are cloned,
// one commit (not 11)
SingleCommitBeforePush: true,
CreateIfNotExists: true,
},
&dummySecret{},
os.Stdout)
require.NoError(t, err)
tree, err := wrap.ReadTree(ctx, "")
require.NoError(t, err)
jj, err := json.MarshalIndent(tree, "", " ")
require.NoError(t, err)
fmt.Printf("TREE:%s\n", string(jj))
ctx = repository.WithAuthorSignature(ctx, repository.CommitSignature{
Name: "xxxxx",
Email: "rrr@yyyy.zzz",
When: time.Now(),
})
for i := 0; i < 10; i++ {
fname := fmt.Sprintf("deep/path/in/test_%d.txt", i)
fmt.Printf("Write:%s\n", fname)
err = wrap.Write(ctx, fname, "", []byte(fmt.Sprintf("body/%d %s", i, time.Now())), "the commit message")
require.NoError(t, err)
}
fmt.Printf("push...\n")
err = wrap.Push(ctx, GoGitPushOptions{
Timeout: 10,
}, os.Stdout)
require.NoError(t, err)
}
pkg/registry/apis/provisioning/repository/local.go
+383
View File
@@ -0,0 +1,383 @@
package repository
import (
"context"
"path"
// Git still uses sha1 for the most part: https://git-scm.com/docs/hash-function-transition
//nolint:gosec
"crypto/sha1"
"encoding/hex"
"errors"
"fmt"
"io"
"io/fs"
"net/http"
"os"
"path/filepath"
"strings"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
type LocalFolderResolver struct {
PermittedPrefixes []string
HomePath string
}
type InvalidLocalFolderError struct {
Path string
AdditionalInfo string
}
var (
_ error = (*InvalidLocalFolderError)(nil)
_ apierrors.APIStatus = (*InvalidLocalFolderError)(nil)
)
func (e *InvalidLocalFolderError) Error() string {
return fmt.Sprintf("the path given ('%s') is invalid for a local repository (%s)", e.Path, e.AdditionalInfo)
}
func (e *InvalidLocalFolderError) Status() metav1.Status {
return metav1.Status{
Status: metav1.StatusFailure,
Code: http.StatusBadRequest,
Reason: metav1.StatusReasonBadRequest,
Message: e.Error(),
}
}
func (r *LocalFolderResolver) LocalPath(p string) (string, error) {
if len(r.PermittedPrefixes) == 0 {
return "", &InvalidLocalFolderError{p, "no permitted prefixes were configured"}
}
originalPath := p
if !path.IsAbs(p) {
p = safepath.Join(r.HomePath, p)
} else {
p = safepath.Clean(p)
}
for _, permitted := range r.PermittedPrefixes {
if safepath.InDir(p, permitted) {
return p, nil
}
}
return "", &InvalidLocalFolderError{originalPath, "the path matches no permitted prefix"}
}
var (
_ Repository = (*localRepository)(nil)
_ Writer = (*localRepository)(nil)
_ Reader = (*localRepository)(nil)
)
type localRepository struct {
config *provisioning.Repository
resolver *LocalFolderResolver
// validated path that can be read if not empty
path string
}
func NewLocal(config *provisioning.Repository, resolver *LocalFolderResolver) *localRepository {
r := &localRepository{
config: config,
resolver: resolver,
}
if config.Spec.Local != nil {
r.path, _ = resolver.LocalPath(config.Spec.Local.Path)
if r.path != "" && !safepath.IsDir(r.path) {
r.path += "/"
}
for i, permitted := range r.resolver.PermittedPrefixes {
r.resolver.PermittedPrefixes[i] = safepath.Clean(permitted)
}
}
return r
}
func (r *localRepository) Config() *provisioning.Repository {
return r.config
}
// Validate implements provisioning.Repository.
func (r *localRepository) Validate() (fields field.ErrorList) {
cfg := r.config.Spec.Local
if cfg == nil {
fields = append(fields, &field.Error{
Type: field.ErrorTypeRequired,
Field: "spec.local",
})
return fields
}
// The path value must be set for local provisioning
if cfg.Path == "" {
fields = append(fields, field.Required(field.NewPath("spec", "local", "path"),
"must enter a path to local file"))
}
// Check if it is valid
_, err := r.resolver.LocalPath(cfg.Path)
if err != nil {
fields = append(fields, field.Invalid(field.NewPath("spec", "local", "path"),
cfg.Path, err.Error()))
}
if err := safepath.IsSafe(cfg.Path); err != nil {
fields = append(fields, field.Invalid(field.NewPath("spec", "local", "path"),
cfg.Path, err.Error()))
}
return fields
}
// Test implements provisioning.Repository.
// NOTE: Validate has been called (and passed) before this function should be called
func (r *localRepository) Test(ctx context.Context) (*provisioning.TestResults, error) {
if r.config.Spec.Local.Path == "" {
return &provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{
"no path is configured",
},
}, nil
}
_, err := r.resolver.LocalPath(r.config.Spec.Local.Path)
if err != nil {
return &provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{
err.Error(),
},
}, nil
}
_, err = os.Stat(r.path)
if errors.Is(err, os.ErrNotExist) {
return &provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{
fmt.Sprintf("directory not found: %s", r.config.Spec.Local.Path),
},
}, nil
}
return &provisioning.TestResults{
Code: http.StatusOK,
Success: true,
}, nil
}
// Test implements provisioning.Repository.
func (r *localRepository) validateRequest(ref string) error {
if ref != "" {
return apierrors.NewBadRequest("local repository does not support ref")
}
if r.path == "" {
_, err := r.resolver.LocalPath(r.config.Spec.Local.Path)
if err != nil {
return err
}
return &apierrors.StatusError{
ErrStatus: metav1.Status{
Message: "the service is missing a root path",
Code: http.StatusFailedDependency,
},
}
}
return nil
}
// ReadResource implements provisioning.Repository.
func (r *localRepository) Read(ctx context.Context, filePath string, ref string) (*FileInfo, error) {
if err := r.validateRequest(ref); err != nil {
return nil, err
}
actualPath := safepath.Join(r.path, filePath)
info, err := os.Stat(actualPath)
if errors.Is(err, os.ErrNotExist) {
return nil, ErrFileNotFound
} else if err != nil {
return nil, fmt.Errorf("stat file: %w", err)
}
if info.IsDir() {
return &FileInfo{
Path: filePath,
Modified: &metav1.Time{
Time: info.ModTime(),
},
}, nil
}
//nolint:gosec
data, err := os.ReadFile(actualPath)
if err != nil {
return nil, fmt.Errorf("read file: %w", err)
}
hash, _, err := r.calculateFileHash(actualPath)
if err != nil {
return nil, fmt.Errorf("calculate hash of file: %w", err)
}
return &FileInfo{
Path: filePath,
Data: data,
Hash: hash,
Modified: &metav1.Time{
Time: info.ModTime(),
},
}, nil
}
// ReadResource implements provisioning.Repository.
func (r *localRepository) ReadTree(ctx context.Context, ref string) ([]FileTreeEntry, error) {
if err := r.validateRequest(ref); err != nil {
return nil, err
}
// Return an empty list when folder does not exist
_, err := os.Stat(r.path)
if errors.Is(err, fs.ErrNotExist) {
return []FileTreeEntry{}, nil
}
rootlen := len(r.path)
entries := make([]FileTreeEntry, 0, 100)
err = filepath.Walk(r.path, func(path string, info fs.FileInfo, err error) error {
if err != nil {
return err
}
entry := FileTreeEntry{
Path: strings.TrimLeft(path[rootlen:], "/"),
Size: info.Size(),
}
if !info.IsDir() {
entry.Blob = true
entry.Hash, _, err = r.calculateFileHash(path)
if err != nil {
return fmt.Errorf("failed to read and calculate hash of path %s: %w", path, err)
}
}
// TODO: do folders have a trailing slash?
entries = append(entries, entry)
return err
})
return entries, err
}
func (r *localRepository) calculateFileHash(path string) (string, int64, error) {
// Treats https://securego.io/docs/rules/g304.html
if !safepath.InDir(path, r.path) {
return "", 0, ErrFileNotFound
}
// We've already made sure the path is safe, so we'll ignore the gosec lint.
//nolint:gosec
file, err := os.OpenFile(path, os.O_RDONLY, 0)
if err != nil {
return "", 0, err
}
// TODO: Define what hashing algorithm we want to use for the entire repository. Maybe a config option?
hasher := sha1.New()
// TODO: context-aware io.Copy? Is that even possible with a reasonable impl?
size, err := io.Copy(hasher, file)
if err != nil {
return "", 0, err
}
// NOTE: EncodeToString (& hex.Encode for that matter) return lower-case hex.
return hex.EncodeToString(hasher.Sum(nil)), size, nil
}
func (r *localRepository) Create(ctx context.Context, fpath string, ref string, data []byte, comment string) error {
if err := r.validateRequest(ref); err != nil {
return err
}
fpath = safepath.Join(r.path, fpath)
_, err := os.Stat(fpath)
if !errors.Is(err, os.ErrNotExist) {
if err != nil {
return apierrors.NewInternalError(fmt.Errorf("failed to check if file exists: %w", err))
}
return apierrors.NewAlreadyExists(provisioning.RepositoryResourceInfo.GroupResource(), fpath)
}
if safepath.IsDir(fpath) {
if data != nil {
return apierrors.NewBadRequest("data cannot be provided for a directory")
}
if err := os.MkdirAll(fpath, 0700); err != nil {
return apierrors.NewInternalError(fmt.Errorf("failed to create path: %w", err))
}
return nil
}
if err := os.MkdirAll(path.Dir(fpath), 0700); err != nil {
return apierrors.NewInternalError(fmt.Errorf("failed to create path: %w", err))
}
return os.WriteFile(fpath, data, 0600)
}
func (r *localRepository) Update(ctx context.Context, path string, ref string, data []byte, comment string) error {
if err := r.validateRequest(ref); err != nil {
return err
}
path = safepath.Join(r.path, path)
if safepath.IsDir(path) {
return apierrors.NewBadRequest("cannot update a directory")
}
if _, err := os.Stat(path); errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("file does not exist")
}
return os.WriteFile(path, data, 0600)
}
func (r *localRepository) Write(ctx context.Context, fpath, ref string, data []byte, comment string) error {
if err := r.validateRequest(ref); err != nil {
return err
}
fpath = safepath.Join(r.path, fpath)
if safepath.IsDir(fpath) {
return os.MkdirAll(fpath, 0700)
}
if err := os.MkdirAll(path.Dir(fpath), 0700); err != nil {
return apierrors.NewInternalError(fmt.Errorf("failed to create path: %w", err))
}
return os.WriteFile(fpath, data, 0600)
}
func (r *localRepository) Delete(ctx context.Context, path string, ref string, comment string) error {
if err := r.validateRequest(ref); err != nil {
return err
}
return os.Remove(safepath.Join(r.path, path))
}
pkg/registry/apis/provisioning/repository/local_test.go
+86
View File
@@ -0,0 +1,86 @@
package repository
import (
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
"github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
)
func TestLocalResolver(t *testing.T) {
resolver := &LocalFolderResolver{
PermittedPrefixes: []string{
"/github/testdata",
},
HomePath: "/",
}
_, err := resolver.LocalPath("github/testdata")
require.NoError(t, err)
_, err = resolver.LocalPath("something")
require.Error(t, err)
}
func TestLocal(t *testing.T) {
// Valid paths test cases
for _, tc := range []struct {
Name string
Path string
PermittedPrefixes []string
ExpectedPath string
}{
{"relative path", "devenv/test", []string{"/home/grafana"}, "/home/grafana/devenv/test/"},
{"absolute path", "/devenv/test", []string{"/devenv"}, "/devenv/test/"},
{"relative path with multiple prefixes", "devenv/test", []string{"/home/grafana", "/devenv"}, "/home/grafana/devenv/test/"},
{"absolute path with multiple prefixes", "/devenv/test", []string{"/home/grafana", "/devenv"}, "/devenv/test/"},
} {
t.Run("valid: "+tc.Name, func(t *testing.T) {
r := NewLocal(&v0alpha1.Repository{
Spec: v0alpha1.RepositorySpec{
Local: &v0alpha1.LocalRepositoryConfig{
Path: tc.Path,
},
},
}, &LocalFolderResolver{PermittedPrefixes: tc.PermittedPrefixes, HomePath: "/home/grafana"})
assert.Equal(t, tc.ExpectedPath, r.path, "expected path to be resolved")
for _, err := range r.Validate() {
assert.Fail(t, "unexpected validation failure", "unexpected validation error on field %s: %s", err.Field, err.ErrorBody())
}
})
}
// Invalid paths test cases
for _, tc := range []struct {
Name string
Path string
PermittedPrefixes []string
}{
{"no configured paths", "invalid/path", nil},
{"path traversal escape", "../../etc/passwd", []string{"/home/grafana"}},
{"unconfigured prefix", "invalid/path", []string{"devenv", "/tmp", "test"}},
} {
t.Run("invalid: "+tc.Name, func(t *testing.T) {
r := NewLocal(&v0alpha1.Repository{
Spec: v0alpha1.RepositorySpec{
Local: &v0alpha1.LocalRepositoryConfig{
Path: tc.Path,
},
},
}, &LocalFolderResolver{PermittedPrefixes: tc.PermittedPrefixes, HomePath: "/home/grafana"})
require.Empty(t, r.path, "no path should be resolved")
errs := r.Validate()
require.NotEmpty(t, errs, "expected validation errors")
for _, err := range errs {
if !assert.Equal(t, "spec.local.path", err.Field) {
assert.FailNow(t, "unexpected validation failure", "unexpected validation error on field %s: %s", err.Field, err.ErrorBody())
}
}
})
}
}
pkg/registry/apis/provisioning/repository/repository.go
+32
View File
@@ -2,6 +2,8 @@ package repository
import (
"context"
"errors"
"fmt"
"io/fs"
"net/http"
@@ -59,6 +61,8 @@ type FileTreeEntry struct {
}
type Reader interface {
Repository
// Read a file from the resource
// This data will be parsed and validated before it is shown to end users
Read(ctx context.Context, path, ref string) (*FileInfo, error)
@@ -72,6 +76,8 @@ type Reader interface {
}
type Writer interface {
Repository
// Write a file to the repository.
// The data has already been validated and is ready for save
Create(ctx context.Context, path, ref string, data []byte, message string) error
@@ -88,8 +94,23 @@ type Writer interface {
Delete(ctx context.Context, path, ref, message string) error
}
type ReaderWriter interface {
Reader
Writer
}
// Hooks called after the repository has been created, updated or deleted
type RepositoryWithURLs interface {
Repository
// Get resource URLs for a file inside a repository
ResourceURLs(ctx context.Context, file *FileInfo) (*provisioning.ResourceURLs, error)
}
// Hooks called after the repository has been created, updated or deleted
type Hooks interface {
Repository
// For repositories that support webhooks
Webhook(ctx context.Context, req *http.Request) (*provisioning.WebhookResponse, error)
OnCreate(ctx context.Context) (*provisioning.WebhookStatus, error)
@@ -126,3 +147,14 @@ type Versioned interface {
LatestRef(ctx context.Context) (string, error)
CompareFiles(ctx context.Context, base, ref string) ([]VersionedFileChange, error)
}
func writeWithReadThenCreateOrUpdate(ctx context.Context, r ReaderWriter, path, ref string, data []byte, comment string) error {
_, err := r.Read(ctx, path, ref)
if err != nil && !(errors.Is(err, ErrFileNotFound)) {
return fmt.Errorf("failed to check if file exists before writing: %w", err)
}
if err == nil {
return r.Update(ctx, path, ref, data, comment)
}
return r.Create(ctx, path, ref, data, comment)
}
pkg/registry/apis/provisioning/repository/repository_mock.go
+295
View File
@@ -0,0 +1,295 @@
// Code generated by mockery v2.50.0. DO NOT EDIT.
package repository
import (
context "context"
http "net/http"
field "k8s.io/apimachinery/pkg/util/validation/field"
mock "github.com/stretchr/testify/mock"
v0alpha1 "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
)
// MockRepository is an autogenerated mock type for the Repository type
type MockRepository struct {
mock.Mock
}
// Config provides a mock function with no fields
func (_m *MockRepository) Config() *v0alpha1.Repository {
ret := _m.Called()
if len(ret) == 0 {
panic("no return value specified for Config")
}
var r0 *v0alpha1.Repository
if rf, ok := ret.Get(0).(func() *v0alpha1.Repository); ok {
r0 = rf()
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*v0alpha1.Repository)
}
}
return r0
}
// Create provides a mock function with given fields: ctx, path, ref, data, message
func (_m *MockRepository) Create(ctx context.Context, path string, ref string, data []byte, message string) error {
ret := _m.Called(ctx, path, ref, data, message)
if len(ret) == 0 {
panic("no return value specified for Create")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, string, string, []byte, string) error); ok {
r0 = rf(ctx, path, ref, data, message)
} else {
r0 = ret.Error(0)
}
return r0
}
// Delete provides a mock function with given fields: ctx, path, ref, message
func (_m *MockRepository) Delete(ctx context.Context, path string, ref string, message string) error {
ret := _m.Called(ctx, path, ref, message)
if len(ret) == 0 {
panic("no return value specified for Delete")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, string, string, string) error); ok {
r0 = rf(ctx, path, ref, message)
} else {
r0 = ret.Error(0)
}
return r0
}
// History provides a mock function with given fields: ctx, path, ref
func (_m *MockRepository) History(ctx context.Context, path string, ref string) ([]v0alpha1.HistoryItem, error) {
ret := _m.Called(ctx, path, ref)
if len(ret) == 0 {
panic("no return value specified for History")
}
var r0 []v0alpha1.HistoryItem
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, string, string) ([]v0alpha1.HistoryItem, error)); ok {
return rf(ctx, path, ref)
}
if rf, ok := ret.Get(0).(func(context.Context, string, string) []v0alpha1.HistoryItem); ok {
r0 = rf(ctx, path, ref)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]v0alpha1.HistoryItem)
}
}
if rf, ok := ret.Get(1).(func(context.Context, string, string) error); ok {
r1 = rf(ctx, path, ref)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Read provides a mock function with given fields: ctx, path, ref
func (_m *MockRepository) Read(ctx context.Context, path string, ref string) (*FileInfo, error) {
ret := _m.Called(ctx, path, ref)
if len(ret) == 0 {
panic("no return value specified for Read")
}
var r0 *FileInfo
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, string, string) (*FileInfo, error)); ok {
return rf(ctx, path, ref)
}
if rf, ok := ret.Get(0).(func(context.Context, string, string) *FileInfo); ok {
r0 = rf(ctx, path, ref)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*FileInfo)
}
}
if rf, ok := ret.Get(1).(func(context.Context, string, string) error); ok {
r1 = rf(ctx, path, ref)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// ReadTree provides a mock function with given fields: ctx, ref
func (_m *MockRepository) ReadTree(ctx context.Context, ref string) ([]FileTreeEntry, error) {
ret := _m.Called(ctx, ref)
if len(ret) == 0 {
panic("no return value specified for ReadTree")
}
var r0 []FileTreeEntry
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, string) ([]FileTreeEntry, error)); ok {
return rf(ctx, ref)
}
if rf, ok := ret.Get(0).(func(context.Context, string) []FileTreeEntry); ok {
r0 = rf(ctx, ref)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).([]FileTreeEntry)
}
}
if rf, ok := ret.Get(1).(func(context.Context, string) error); ok {
r1 = rf(ctx, ref)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Test provides a mock function with given fields: ctx
func (_m *MockRepository) Test(ctx context.Context) (*v0alpha1.TestResults, error) {
ret := _m.Called(ctx)
if len(ret) == 0 {
panic("no return value specified for Test")
}
var r0 *v0alpha1.TestResults
var r1 error
if rf, ok := ret.Get(0).(func(context.Context) (*v0alpha1.TestResults, error)); ok {
return rf(ctx)
}
if rf, ok := ret.Get(0).(func(context.Context) *v0alpha1.TestResults); ok {
r0 = rf(ctx)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*v0alpha1.TestResults)
}
}
if rf, ok := ret.Get(1).(func(context.Context) error); ok {
r1 = rf(ctx)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Update provides a mock function with given fields: ctx, path, ref, data, message
func (_m *MockRepository) Update(ctx context.Context, path string, ref string, data []byte, message string) error {
ret := _m.Called(ctx, path, ref, data, message)
if len(ret) == 0 {
panic("no return value specified for Update")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, string, string, []byte, string) error); ok {
r0 = rf(ctx, path, ref, data, message)
} else {
r0 = ret.Error(0)
}
return r0
}
// Validate provides a mock function with no fields
func (_m *MockRepository) Validate() field.ErrorList {
ret := _m.Called()
if len(ret) == 0 {
panic("no return value specified for Validate")
}
var r0 field.ErrorList
if rf, ok := ret.Get(0).(func() field.ErrorList); ok {
r0 = rf()
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(field.ErrorList)
}
}
return r0
}
// Webhook provides a mock function with given fields: ctx, req
func (_m *MockRepository) Webhook(ctx context.Context, req *http.Request) (*v0alpha1.WebhookResponse, error) {
ret := _m.Called(ctx, req)
if len(ret) == 0 {
panic("no return value specified for Webhook")
}
var r0 *v0alpha1.WebhookResponse
var r1 error
if rf, ok := ret.Get(0).(func(context.Context, *http.Request) (*v0alpha1.WebhookResponse, error)); ok {
return rf(ctx, req)
}
if rf, ok := ret.Get(0).(func(context.Context, *http.Request) *v0alpha1.WebhookResponse); ok {
r0 = rf(ctx, req)
} else {
if ret.Get(0) != nil {
r0 = ret.Get(0).(*v0alpha1.WebhookResponse)
}
}
if rf, ok := ret.Get(1).(func(context.Context, *http.Request) error); ok {
r1 = rf(ctx, req)
} else {
r1 = ret.Error(1)
}
return r0, r1
}
// Write provides a mock function with given fields: ctx, path, ref, data, message
func (_m *MockRepository) Write(ctx context.Context, path string, ref string, data []byte, message string) error {
ret := _m.Called(ctx, path, ref, data, message)
if len(ret) == 0 {
panic("no return value specified for Write")
}
var r0 error
if rf, ok := ret.Get(0).(func(context.Context, string, string, []byte, string) error); ok {
r0 = rf(ctx, path, ref, data, message)
} else {
r0 = ret.Error(0)
}
return r0
}
// NewMockRepository creates a new instance of MockRepository. It also registers a testing interface on the mock and a cleanup function to assert the mocks expectations.
// The first argument is typically a *testing.T value.
func NewMockRepository(t interface {
mock.TestingT
Cleanup(func())
}) *MockRepository {
mock := &MockRepository{}
mock.Mock.Test(t)
t.Cleanup(func() { mock.AssertExpectations(t) })
return mock
}
pkg/registry/apis/provisioning/repository/test.go
+87
View File
@@ -0,0 +1,87 @@
package repository
import (
"context"
"fmt"
"net/http"
"slices"
"k8s.io/apimachinery/pkg/util/validation/field"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
)
// Tester is a struct that implements the Tester interface
// it's temporary
// FIXME: remove as soon as controller and jobs refactoring PRs are merged
type Tester struct{}
func (t *Tester) TestRepository(ctx context.Context, repo Repository) (*provisioning.TestResults, error) {
return TestRepository(ctx, repo)
}
func TestRepository(ctx context.Context, repo Repository) (*provisioning.TestResults, error) {
errors := ValidateRepository(repo)
if len(errors) > 0 {
rsp := &provisioning.TestResults{
Code: http.StatusUnprocessableEntity, // Invalid
Success: false,
Errors: make([]string, len(errors)),
}
for i, v := range errors {
rsp.Errors[i] = v.Error()
}
return rsp, nil
}
return repo.Test(ctx)
}
func ValidateRepository(repo Repository) field.ErrorList {
list := repo.Validate()
cfg := repo.Config()
if cfg.Spec.Title == "" {
list = append(list, field.Required(field.NewPath("spec", "title"), "a repository title must be given"))
}
if cfg.Spec.Sync.Enabled && cfg.Spec.Sync.Target == "" {
list = append(list, field.Required(field.NewPath("spec", "sync", "target"),
"The target type is required when sync is enabled"))
}
if cfg.Spec.Sync.Enabled && cfg.Spec.Sync.IntervalSeconds < 10 {
list = append(list, field.Invalid(field.NewPath("spec", "sync", "intervalSeconds"),
cfg.Spec.Sync.IntervalSeconds, fmt.Sprintf("Interval must be at least %d seconds", 10)))
}
// Reserved names (for now)
reserved := []string{"classic", "sql", "SQL", "plugins", "legacy", "new", "job", "github", "s3", "gcs", "file", "new", "create", "update", "delete"}
if slices.Contains(reserved, cfg.Name) {
list = append(list, field.Invalid(field.NewPath("metadata", "name"), cfg.Name, "Name is reserved, choose a different identifier"))
}
if cfg.Spec.Type != provisioning.LocalRepositoryType && cfg.Spec.Local != nil {
list = append(list, field.Invalid(field.NewPath("spec", "local"),
cfg.Spec.GitHub, "Local config only valid when type is local"))
}
if cfg.Spec.Type != provisioning.GitHubRepositoryType && cfg.Spec.GitHub != nil {
list = append(list, field.Invalid(field.NewPath("spec", "github"),
cfg.Spec.GitHub, "Github config only valid when type is github"))
}
for _, w := range cfg.Spec.Workflows {
switch w {
case provisioning.WriteWorkflow: // valid; no fall thru
case provisioning.BranchWorkflow:
if cfg.Spec.Type != provisioning.GitHubRepositoryType {
list = append(list, field.Invalid(field.NewPath("spec", "workflow"), w, "branch is only supported on git repositories"))
}
default:
list = append(list, field.Invalid(field.NewPath("spec", "workflow"), w, "invalid workflow"))
}
}
return list
}
pkg/registry/apis/provisioning/repository/test_test.go
+315
View File
@@ -0,0 +1,315 @@
package repository
import (
"context"
"fmt"
"net/http"
"testing"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/stretchr/testify/mock"
"github.com/stretchr/testify/require"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/util/validation/field"
)
func TestValidateRepository(t *testing.T) {
tests := []struct {
name string
repository *MockRepository
expectedErrs int
validateError func(t *testing.T, errors field.ErrorList)
}{
{
name: "valid repository",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 0,
},
{
name: "missing title",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "spec.title: Required value")
},
},
{
name: "sync enabled without target",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
Sync: provisioning.SyncOptions{
Enabled: true,
IntervalSeconds: 10,
},
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "spec.sync.target: Required value")
},
},
{
name: "sync interval too low",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
Sync: provisioning.SyncOptions{
Enabled: true,
Target: "test",
IntervalSeconds: 5,
},
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "spec.sync.intervalSeconds: Invalid value")
},
},
{
name: "reserved name",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
ObjectMeta: metav1.ObjectMeta{
Name: "sql",
},
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "metadata.name: Invalid value")
},
},
{
name: "mismatched local config",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
Type: provisioning.GitHubRepositoryType,
Local: &provisioning.LocalRepositoryConfig{},
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "spec.local: Invalid value")
},
},
{
name: "mismatched github config",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
Type: provisioning.LocalRepositoryType,
GitHub: &provisioning.GitHubRepositoryConfig{},
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 1,
validateError: func(t *testing.T, errors field.ErrorList) {
require.Contains(t, errors.ToAggregate().Error(), "spec.github: Invalid value")
},
},
{
name: "multiple validation errors",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
ObjectMeta: metav1.ObjectMeta{
Name: "sql",
},
Spec: provisioning.RepositorySpec{
Sync: provisioning.SyncOptions{
Enabled: true,
IntervalSeconds: 5,
},
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedErrs: 4, // Updated from 3 to 4 to match actual errors:
// 1. missing title
// 2. sync target missing
// 3. sync interval too low
// 4. reserved name
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
errors := ValidateRepository(tt.repository)
require.Len(t, errors, tt.expectedErrs)
if tt.validateError != nil {
tt.validateError(t, errors)
}
})
}
}
func TestTestRepository(t *testing.T) {
tests := []struct {
name string
repository *MockRepository
expectedCode int
expectedErrs []string
expectedError error
}{
{
name: "validation fails",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
// Missing required title
},
})
m.On("Validate").Return(field.ErrorList{})
return m
}(),
expectedCode: http.StatusUnprocessableEntity,
expectedErrs: []string{"spec.title: Required value: a repository title must be given"},
},
{
name: "test passes",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
m.On("Validate").Return(field.ErrorList{})
m.On("Test", mock.Anything).Return(&provisioning.TestResults{
Code: http.StatusOK,
Success: true,
}, nil)
return m
}(),
expectedCode: http.StatusOK,
expectedErrs: nil,
},
{
name: "test fails with error",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
m.On("Validate").Return(field.ErrorList{})
m.On("Test", mock.Anything).Return(nil, fmt.Errorf("test error"))
return m
}(),
expectedError: fmt.Errorf("test error"),
},
{
name: "test fails with results",
repository: func() *MockRepository {
m := NewMockRepository(t)
m.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
m.On("Validate").Return(field.ErrorList{})
m.On("Test", mock.Anything).Return(&provisioning.TestResults{
Code: http.StatusBadRequest,
Success: false,
Errors: []string{"test failed"},
}, nil)
return m
}(),
expectedCode: http.StatusBadRequest,
expectedErrs: []string{"test failed"},
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
results, err := TestRepository(context.Background(), tt.repository)
if tt.expectedError != nil {
require.Error(t, err)
require.Equal(t, tt.expectedError.Error(), err.Error())
return
}
require.NoError(t, err)
require.NotNil(t, results)
require.Equal(t, tt.expectedCode, results.Code)
if tt.expectedErrs != nil {
require.Equal(t, tt.expectedErrs, results.Errors)
require.False(t, results.Success)
} else {
require.True(t, results.Success)
require.Empty(t, results.Errors)
}
})
}
}
func TestTester_TestRepository(t *testing.T) {
tester := &Tester{}
// Test that it properly delegates to TestRepository
repository := NewMockRepository(t)
repository.On("Config").Return(&provisioning.Repository{
Spec: provisioning.RepositorySpec{
Title: "Test Repo",
},
})
repository.On("Validate").Return(field.ErrorList{})
repository.On("Test", mock.Anything).Return(&provisioning.TestResults{
Code: http.StatusOK,
Success: true,
}, nil)
results, err := tester.TestRepository(context.Background(), repository)
require.NoError(t, err)
require.NotNil(t, results)
require.Equal(t, http.StatusOK, results.Code)
require.True(t, results.Success)
}
pkg/registry/apis/provisioning/repository/workflows.go
+40
View File
@@ -0,0 +1,40 @@
package repository
import (
"net/http"
apierrors "k8s.io/apimachinery/pkg/api/errors"
v1 "k8s.io/apimachinery/pkg/apis/meta/v1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
)
func IsWriteAllowed(repo *provisioning.Repository, ref string) error {
if len(repo.Spec.Workflows) == 0 {
return apierrors.NewBadRequest("this repository is read only")
}
if ref == "" {
for _, v := range repo.Spec.Workflows {
if v == provisioning.WriteWorkflow {
return nil // found
}
}
return apierrors.NewBadRequest("this repository does not support the write workflow")
}
// Only github
if repo.Spec.Type != provisioning.GitHubRepositoryType {
return &apierrors.StatusError{ErrStatus: v1.Status{
Code: http.StatusPreconditionFailed,
Message: "Only github supports writing to a branch",
}}
}
for _, v := range repo.Spec.Workflows {
if v == provisioning.BranchWorkflow {
return nil
}
}
return apierrors.NewBadRequest("this repository does not support the branch workflow")
}
pkg/registry/apis/provisioning/request.go
+69
View File
@@ -0,0 +1,69 @@
package provisioning
import (
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
)
const (
// contentTypeJSON is the Content-Type for JSON requests as go standard library does not provide one
contentTypeJSON = "application/json"
// defaultMaxBodySize is the default max size for request bodies (10KB)
defaultMaxBodySize = 10 * 1024
// errMsgRequestTooLarge is the error message for request bodies that are too large
errMsgRequestTooLarge = "request body too large"
)
// readBody reads the request body and limits the size
func readBody(r *http.Request, maxSize int64) ([]byte, error) {
limitedBody := http.MaxBytesReader(nil, r.Body, maxSize)
body, err := io.ReadAll(limitedBody)
if err != nil {
var maxBytesError *http.MaxBytesError
if errors.As(err, &maxBytesError) {
return nil, fmt.Errorf("%s: max size %d bytes", errMsgRequestTooLarge, maxSize)
}
return nil, fmt.Errorf("error reading request body: %w", err)
}
return body, nil
}
// isJSONContentType checks if the request has the JSON Content-Type
func isJSONContentType(r *http.Request) bool {
contentType := r.Header.Get("Content-Type")
return strings.HasPrefix(contentType, contentTypeJSON)
}
// unmarshalJSON unmarshals the request body into the provided interface
// it also checks the Content-Type and limits the size of the request body
func unmarshalJSON(r *http.Request, maxSize int64, v interface{}) error {
if !isJSONContentType(r) {
return fmt.Errorf("content type is not JSON: %s", r.Header.Get("Content-Type"))
}
r.Body = http.MaxBytesReader(nil, r.Body, maxSize)
decoder := json.NewDecoder(r.Body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(v); err != nil {
var maxBytesError *http.MaxBytesError
if errors.As(err, &maxBytesError) {
return fmt.Errorf("%s: max size %d bytes", errMsgRequestTooLarge, maxSize)
}
if err == io.EOF {
return fmt.Errorf("empty request body")
}
return fmt.Errorf("error decoding JSON: %w", err)
}
if decoder.More() {
return fmt.Errorf("multiple JSON objects not allowed")
}
return nil
}
pkg/registry/apis/provisioning/request_test.go
+176
View File
@@ -0,0 +1,176 @@
package provisioning
import (
"net/http/httptest"
"strings"
"testing"
)
func TestReadBody(t *testing.T) {
tests := []struct {
name string
body string
maxSize int64
wantErr bool
errContains string
}{
{
name: "valid small body",
body: "hello",
maxSize: 10,
},
{
name: "body too large",
body: "this is a very long body that exceeds the limit",
maxSize: 10,
wantErr: true,
errContains: errMsgRequestTooLarge,
},
{
name: "body exactly at limit",
body: "1234567890",
maxSize: 10,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := httptest.NewRequest("POST", "/", strings.NewReader(tt.body))
got, err := readBody(r, tt.maxSize)
if tt.wantErr {
if err == nil {
t.Error("readBody() expected error but got none")
}
if !strings.Contains(err.Error(), tt.errContains) {
t.Errorf("readBody() error = %v, want containing %v", err, tt.errContains)
}
return
}
if err != nil {
t.Errorf("readBody() unexpected error = %v", err)
return
}
if string(got) != tt.body {
t.Errorf("readBody() = %v, want %v", string(got), tt.body)
}
})
}
}
func TestIsJSONContentType(t *testing.T) {
tests := []struct {
name string
contentType string
want bool
}{
{
name: "valid JSON content type",
contentType: "application/json",
want: true,
},
{
name: "JSON with charset",
contentType: "application/json; charset=utf-8",
want: true,
},
{
name: "not JSON",
contentType: "text/plain",
want: false,
},
{
name: "empty content type",
contentType: "",
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := httptest.NewRequest("POST", "/", nil)
r.Header.Set("Content-Type", tt.contentType)
if got := isJSONContentType(r); got != tt.want {
t.Errorf("isJSONContentType() = %v, want %v", got, tt.want)
}
})
}
}
func TestUnmarshalJSON(t *testing.T) {
type testStruct struct {
Name string `json:"name"`
Age int `json:"age"`
}
tests := []struct {
name string
body string
maxSize int64
contentType string
wantErr bool
errContains string
}{
{
name: "valid JSON",
body: `{"name":"test","age":30}`,
maxSize: 1024,
contentType: contentTypeJSON,
},
{
name: "invalid content type",
body: `{"name":"test"}`,
maxSize: 1024,
contentType: "text/plain",
wantErr: true,
errContains: "content type is not JSON",
},
{
name: "body too large",
body: `{"name":"test","age":30}`,
maxSize: 10,
contentType: contentTypeJSON,
wantErr: true,
errContains: errMsgRequestTooLarge,
},
{
name: "multiple JSON objects",
body: `{"name":"test"} {"name":"test2"}`,
maxSize: 1024,
contentType: contentTypeJSON,
wantErr: true,
errContains: "multiple JSON objects not allowed",
},
{
name: "empty body",
body: "",
maxSize: 1024,
contentType: contentTypeJSON,
wantErr: true,
errContains: "empty request body",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
r := httptest.NewRequest("POST", "/", strings.NewReader(tt.body))
r.Header.Set("Content-Type", tt.contentType)
var result testStruct
err := unmarshalJSON(r, tt.maxSize, &result)
if tt.wantErr {
if err == nil {
t.Error("unmarshalJSON() expected error but got none")
}
if !strings.Contains(err.Error(), tt.errContains) {
t.Errorf("unmarshalJSON() error = %v, want containing %v", err, tt.errContains)
}
return
}
if err != nil {
t.Errorf("unmarshalJSON() unexpected error = %v", err)
}
})
}
}
pkg/registry/apis/provisioning/resources/client.go
+167
View File
@@ -0,0 +1,167 @@
package resources
import (
"context"
"fmt"
"sync"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/client-go/dynamic"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
iam "github.com/grafana/grafana/pkg/apis/iam/v0alpha1"
"github.com/grafana/grafana/pkg/services/apiserver"
"github.com/grafana/grafana/pkg/services/apiserver/client"
)
type ClientFactory struct {
configProvider apiserver.RestConfigProvider
}
func NewClientFactory(configProvider apiserver.RestConfigProvider) *ClientFactory {
return &ClientFactory{configProvider}
}
func (f *ClientFactory) Clients(ctx context.Context, namespace string) (*ResourceClients, error) {
restConfig, err := f.configProvider.GetRestConfig(ctx)
if err != nil {
return nil, err
}
if namespace == "" {
return nil, fmt.Errorf("missing namespace")
}
discovery, err := client.NewDiscoveryClient(restConfig)
if err != nil {
return nil, err
}
client, err := dynamic.NewForConfig(restConfig)
if err != nil {
return nil, err
}
return &ResourceClients{
namespace: namespace,
discovery: discovery,
dynamic: client,
byKind: make(map[schema.GroupVersionKind]*clientInfo),
byResource: make(map[schema.GroupVersionResource]*clientInfo),
}, nil
}
type ResourceClients struct {
namespace string
dynamic dynamic.Interface
discovery client.DiscoveryClient
// ResourceInterface cache for this context + namespace
mutex sync.Mutex
byKind map[schema.GroupVersionKind]*clientInfo
byResource map[schema.GroupVersionResource]*clientInfo
}
type clientInfo struct {
gvk schema.GroupVersionKind
gvr schema.GroupVersionResource
client dynamic.ResourceInterface
}
func (c *ResourceClients) ForKind(gvk schema.GroupVersionKind) (dynamic.ResourceInterface, schema.GroupVersionResource, error) {
c.mutex.Lock()
defer c.mutex.Unlock()
info, ok := c.byKind[gvk]
if ok && info.client != nil {
return info.client, info.gvr, nil
}
gvr, err := c.discovery.GetResourceForKind(gvk)
if err != nil {
return nil, schema.GroupVersionResource{}, err
}
info = &clientInfo{
gvk: gvk,
gvr: gvr,
client: c.dynamic.Resource(gvr).Namespace(c.namespace),
}
c.byKind[gvk] = info
c.byResource[gvr] = info
return info.client, info.gvr, nil
}
func (c *ResourceClients) ForResource(gvr schema.GroupVersionResource) (dynamic.ResourceInterface, schema.GroupVersionKind, error) {
c.mutex.Lock()
defer c.mutex.Unlock()
info, ok := c.byResource[gvr]
if ok && info.client != nil {
return info.client, info.gvk, nil
}
var err error
var gvk schema.GroupVersionKind
var versionless schema.GroupVersionResource
if gvr.Version == "" {
versionless = gvr
gvr, gvk, err = c.discovery.GetPreferredVesion(schema.GroupResource{
Group: gvr.Group,
Resource: gvr.Resource,
})
if err != nil {
return nil, schema.GroupVersionKind{}, err
}
info, ok := c.byResource[gvr]
if ok && info.client != nil {
c.byResource[versionless] = info
return info.client, info.gvk, nil
}
} else {
gvk, err = c.discovery.GetKindForResource(gvr)
if err != nil {
return nil, schema.GroupVersionKind{}, err
}
}
info = &clientInfo{
gvk: gvk,
gvr: gvr,
client: c.dynamic.Resource(gvr).Namespace(c.namespace),
}
c.byKind[gvk] = info
c.byResource[gvr] = info
if versionless.Group != "" {
c.byResource[versionless] = info
}
return info.client, info.gvk, nil
}
func (c *ResourceClients) Folder() (dynamic.ResourceInterface, error) {
v, _, err := c.ForResource(schema.GroupVersionResource{
Group: folders.GROUP,
Version: folders.VERSION,
Resource: folders.RESOURCE,
})
return v, err
}
func (c *ResourceClients) User() (dynamic.ResourceInterface, error) {
v, _, err := c.ForResource(schema.GroupVersionResource{
Group: iam.GROUP,
Version: iam.VERSION,
Resource: iam.UserResourceInfo.GroupResource().Resource,
})
return v, err
}
func (c *ResourceClients) Dashboard() (dynamic.ResourceInterface, error) {
v, _, err := c.ForResource(schema.GroupVersionResource{
Group: dashboard.GROUP,
Version: dashboard.VERSION,
Resource: dashboard.DASHBOARD_RESOURCE,
})
return v, err
}
pkg/registry/apis/provisioning/resources/fileformat.go
+107
View File
@@ -0,0 +1,107 @@
package resources
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/apimachinery/pkg/runtime/serializer/yaml"
"github.com/grafana/grafana-app-sdk/logging"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
var ErrUnableToReadResourceBytes = errors.New("unable to read bytes as a resource")
var ErrClassicResourceIsAlreadyK8sForm = errors.New("classic resource is already structured with apiVersion and kind")
// This reads a "classic" file format and will convert it to an unstructured k8s resource
// The file path may determine how the resource is parsed
//
// The context and logger are both only used for logging purposes. They do not control any logic.
func ReadClassicResource(ctx context.Context, info *repository.FileInfo) (*unstructured.Unstructured, *schema.GroupVersionKind, provisioning.ClassicFileType, error) {
var value map[string]any
// Try parsing as JSON
if info.Data[0] == '{' {
err := json.Unmarshal(info.Data, &value)
if err != nil {
return nil, nil, "", err
}
} else {
return nil, nil, "", fmt.Errorf("yaml not yet implemented")
}
// regular version headers exist
// TODO: do we intend on this checking Kind or kind? document reasoning.
if value["apiVersion"] != nil {
if value["kind"] != nil {
return nil, nil, "", ErrClassicResourceIsAlreadyK8sForm
}
logging.FromContext(ctx).Debug("TODO... likely a provisioning",
"apiVersion", value["apiVersion"],
"kind", value["Kind"])
gv, err := schema.ParseGroupVersion(value["apiVersion"].(string))
if err != nil {
return nil, nil, "", fmt.Errorf("invalid apiVersion")
}
gvk := gv.WithKind(value["Kind"].(string))
return &unstructured.Unstructured{Object: value}, &gvk, "", nil
}
// If this is a dashboard, convert it
if value["panels"] != nil &&
value["schemaVersion"] != nil &&
value["tags"] != nil {
gvk := &schema.GroupVersionKind{
Group: dashboard.GROUP,
Version: dashboard.VERSION, // v1
Kind: "Dashboard"}
return &unstructured.Unstructured{
Object: map[string]interface{}{
"apiVersion": gvk.GroupVersion().String(),
"kind": gvk.Kind,
"metadata": map[string]any{
"name": value["uid"],
},
"spec": value,
},
}, gvk, provisioning.ClassicDashboard, nil
}
return nil, nil, "", ErrUnableToReadResourceBytes
}
// DecodeYAMLObject reads the input as YAML and outputs its Kubernetes resource, if it is one.
// Note that all JSON is also valid YAML, so this can also be used for JSON data.
func DecodeYAMLObject(input io.Reader) (*unstructured.Unstructured, *schema.GroupVersionKind, error) {
data, err := io.ReadAll(input)
if err != nil {
return nil, nil, err
}
obj, gvk, err := yaml.NewDecodingSerializer(unstructured.UnstructuredJSONScheme).
Decode(data, nil, nil)
if err != nil {
return nil, gvk, err
}
// The decoder should put it directly into an unstructured object
val, ok := obj.(*unstructured.Unstructured)
if ok {
return val, gvk, err
}
unstructuredMap, err := runtime.DefaultUnstructuredConverter.ToUnstructured(obj)
if err != nil {
return nil, gvk, err
}
return &unstructured.Unstructured{Object: unstructuredMap}, gvk, err
}
pkg/registry/apis/provisioning/resources/fileformat_test.go
+118
View File
@@ -0,0 +1,118 @@
package resources
import (
"bytes"
"context"
"os"
"path"
"testing"
"github.com/stretchr/testify/require"
"k8s.io/apimachinery/pkg/runtime/schema"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
func TestUtils(t *testing.T) {
t.Run("load playlist json", func(t *testing.T) {
obj, gvk, err := DecodeYAMLObject(bytes.NewReader([]byte(`{
"kind": "Playlist",
"apiVersion": "playlist.grafana.app/v0alpha1",
"metadata": {
"name": "hello"
},
"spec": {
"title": "Playlist from provisioning"
}
}`)))
require.NoError(t, err)
require.Equal(t, &schema.GroupVersionKind{
Group: "playlist.grafana.app",
Version: "v0alpha1",
Kind: "Playlist",
}, gvk)
require.NotNil(t, obj)
})
t.Run("YAML Parsing", func(t *testing.T) {
obj, gvk, err := DecodeYAMLObject(bytes.NewReader([]byte("kind: xyz\npi: 3.1415")))
require.NoError(t, err)
require.NotNil(t, gvk)
require.Equal(t, "xyz", obj.Object["kind"])
require.Equal(t, 3.1415, obj.Object["pi"])
// // Tabs in the value
// _, _, err = LoadYAMLOrJSON(bytes.NewReader([]byte("kind: xyz\n\tpi: 3.1415")))
// require.Equal(t, ErrYamlContainsTabs, err)
})
t.Run("load playlist yaml", func(t *testing.T) {
obj, gvk, err := DecodeYAMLObject(bytes.NewReader([]byte(`
apiVersion: playlist.grafana.app/v0alpha1
kind: Playlist
metadata:
name: hello
spec:
title: a title
`)))
require.NoError(t, err)
require.Equal(t, &schema.GroupVersionKind{
Group: "playlist.grafana.app",
Version: "v0alpha1",
Kind: "Playlist",
}, gvk)
require.NotNil(t, obj)
})
t.Run("load dashboard json", func(t *testing.T) {
// Support dashboard conversion
obj, gvk, classic, err := ReadClassicResource(context.Background(), &repository.FileInfo{
Data: []byte(`{
"schemaVersion": 7,
"panels": [],
"tags": []
}`),
})
require.NoError(t, err)
require.Equal(t, provisioning.ClassicDashboard, classic)
require.Equal(t, &schema.GroupVersionKind{
Group: "dashboard.grafana.app",
Version: "v0alpha1",
Kind: "Dashboard",
}, gvk)
require.NotNil(t, obj)
})
t.Run("lint dashboard", func(t *testing.T) {
var err error
info := &repository.FileInfo{
Path: "devenv/dev-dashboards/panel-timeline/timeline-demo.json",
}
info.Data, err = os.ReadFile(path.Join("../../../../..", info.Path))
require.NoError(t, err)
parser := &Parser{
repo: provisioning.ResourceRepositoryInfo{
Name: "test",
},
}
// try to validate (and lint)
validate := true
// Support dashboard conversion
parsed, err := parser.Parse(context.Background(), info, validate)
require.Error(t, err) // no clients configured!
require.Equal(t, provisioning.ClassicDashboard, parsed.Classic)
require.Equal(t, &schema.GroupVersionKind{
Group: "dashboard.grafana.app",
Version: "v0alpha1",
Kind: "Dashboard",
}, parsed.GVK)
})
}
pkg/registry/apis/provisioning/resources/filepath.go
+42
View File
@@ -0,0 +1,42 @@
package resources
import (
"errors"
"path"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
var (
ErrPathTooDeep = errors.New("the path is too deep")
ErrUnsupportedFileExtension = errors.New("unsupported file extension")
ErrNotRelative = errors.New("path must be relative to the root")
)
const maxPathDepth = 8
// IsPathSupported checks if the file path is supported by the provisioning API.
// it also validates if the path is safe and if the file extension is supported.
func IsPathSupported(filePath string) error {
// Validate the path for any traversal attempts first
if err := safepath.IsSafe(filePath); err != nil {
return err
}
if safepath.Depth(filePath) > maxPathDepth {
return ErrPathTooDeep
}
if safepath.IsAbs(filePath) {
return ErrNotRelative
}
// Only check file extension if it's not a folder path
if !safepath.IsDir(filePath) {
if ext := path.Ext(filePath); ext != ".yml" && ext != ".yaml" && ext != ".json" {
return ErrUnsupportedFileExtension
}
}
return nil
}
pkg/registry/apis/provisioning/resources/filepath_test.go
+72
View File
@@ -0,0 +1,72 @@
package resources
import (
"testing"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
"github.com/stretchr/testify/require"
)
func TestIsPathSupported(t *testing.T) {
tests := []struct {
name string
path string
expectedErr error
}{
{
name: "valid yaml file",
path: "dashboards/my-dashboard.yaml",
},
{
name: "valid yml file",
path: "dashboards/my-dashboard.yml",
},
{
name: "valid json file",
path: "dashboards/my-dashboard.json",
},
{
name: "valid nested path",
path: "dashboards/folder1/folder2/my-dashboard.yaml",
},
{
name: "valid directory path",
path: "dashboards/folder1/",
},
{
name: "unsupported file extension",
path: "dashboards/my-dashboard.txt",
expectedErr: ErrUnsupportedFileExtension,
},
{
name: "path traversal attempt",
path: "../dashboards/my-dashboard.yaml",
expectedErr: safepath.ErrPathTraversalAttempt,
},
{
name: "path too deep",
path: "level1/level2/level3/level4/level5/level6/level7/level8/level9/dashboard.yaml",
expectedErr: ErrPathTooDeep,
},
{
name: "absolute path",
path: "/etc/dashboards/my-dashboard.yaml",
expectedErr: ErrNotRelative,
},
{
name: "empty directory path",
path: "",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := IsPathSupported(tt.path)
if tt.expectedErr != nil {
require.ErrorIs(t, err, tt.expectedErr)
} else {
require.NoError(t, err)
}
})
}
}
pkg/registry/apis/provisioning/resources/folders.go
+133
View File
@@ -0,0 +1,133 @@
package resources
import (
"context"
"fmt"
apierrors "k8s.io/apimachinery/pkg/api/errors"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/client-go/dynamic"
"github.com/grafana/grafana/pkg/apimachinery/utils"
"github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
type FolderManager struct {
repo repository.Repository
lookup *FolderTree
client dynamic.ResourceInterface
}
func NewFolderManager(repo repository.Repository, client dynamic.ResourceInterface) *FolderManager {
return &FolderManager{
repo: repo,
lookup: NewEmptyFolderTree(),
client: client,
}
}
func (fm *FolderManager) Client() dynamic.ResourceInterface {
return fm.client
}
// EnsureFoldersExist creates the folder structure in the cluster.
func (fm *FolderManager) EnsureFolderPathExist(ctx context.Context, filePath string) (parent string, err error) {
cfg := fm.repo.Config()
parent = RootFolder(cfg)
dir := filePath
if !safepath.IsDir(filePath) {
dir = safepath.Dir(filePath)
}
if dir == "" {
return parent, nil
}
f := ParseFolder(dir, cfg.Name)
if fm.lookup.In(f.ID) {
return f.ID, nil
}
err = safepath.Walk(ctx, f.Path, func(ctx context.Context, traverse string) error {
f := ParseFolder(traverse, cfg.GetName())
if fm.lookup.In(f.ID) {
parent = f.ID
return nil
}
if err := fm.EnsureFolderExists(ctx, f, parent); err != nil {
return fmt.Errorf("ensure folder exists: %w", err)
}
fm.lookup.Add(f, parent)
parent = f.ID
return nil
})
if err != nil {
return "", err
}
return f.ID, nil
}
// EnsureFolderExists creates the folder if it doesn't exist.
// If the folder already exists:
// - it will error if the folder is not owned by this repository
func (fm *FolderManager) EnsureFolderExists(ctx context.Context, folder Folder, parent string) error {
cfg := fm.repo.Config()
obj, err := fm.client.Get(ctx, folder.ID, metav1.GetOptions{})
if err == nil {
current, ok := obj.GetAnnotations()[utils.AnnoKeyManagerIdentity]
if !ok {
return fmt.Errorf("target folder is not managed by a repository")
}
if current != cfg.Name {
return fmt.Errorf("target folder is managed by a different repository (%s)", current)
}
return nil
} else if !apierrors.IsNotFound(err) {
return fmt.Errorf("failed to check if folder exists: %w", err)
}
obj = &unstructured.Unstructured{
Object: map[string]interface{}{
"spec": map[string]any{
"title": folder.Title,
},
},
}
obj.SetAPIVersion(v0alpha1.APIVERSION)
obj.SetKind(v0alpha1.FolderResourceInfo.GroupVersionKind().Kind)
obj.SetNamespace(cfg.GetNamespace())
obj.SetName(folder.ID)
meta, err := utils.MetaAccessor(obj)
if err != nil {
return fmt.Errorf("create meta accessor for the object: %w", err)
}
if parent != "" {
meta.SetFolder(parent)
}
meta.SetManagerProperties(utils.ManagerProperties{
Kind: utils.ManagerKindRepo,
Identity: cfg.GetName(),
})
meta.SetSourceProperties(utils.SourceProperties{
Path: folder.Path,
})
if _, err := fm.client.Create(ctx, obj, metav1.CreateOptions{}); err != nil {
return fmt.Errorf("failed to create folder: %w", err)
}
return nil
}
func (fm *FolderManager) GetFolder(ctx context.Context, name string) (*unstructured.Unstructured, error) {
return fm.client.Get(ctx, name, metav1.GetOptions{})
}
pkg/registry/apis/provisioning/resources/id.go
+141
View File
@@ -0,0 +1,141 @@
package resources
import (
"crypto/sha256"
"encoding/base64"
"strings"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
// sanitiseKubeName removes all characters that don't fulfil the DNS subdomain name rules: https://kubernetes.io/docs/concepts/overview/working-with-objects/names/#dns-subdomain-names
// > contain no more than 253 characters
// > contain only lowercase alphanumeric characters, '-' or '.'
// > start with an alphanumeric character
// > end with an alphanumeric character
//
// That said, the Kubernetes name is extended with one more rule: it must not contain more than 40 characters.
// We do this to support storage modes that write to legacy storage, where UIDs are limited to 40 characters or less.
//
// If no characters are valid, this returns an empty string.
func sanitiseKubeName(s string) string {
// Note: Builder never returns an error.
var b strings.Builder
lastHyphen := false // Having at most 1 hyphen in a row is not a requirement, but is closer to standard convention.
for _, r := range strings.ToLower(s) {
if (r >= 'a' && r <= 'z') || (r >= '0' && r <= '9') || r == '.' || r == '-' {
// We don't want to have multiple hyphens following one another.
if !lastHyphen || r != '-' {
lastHyphen = r == '-'
_, _ = b.WriteRune(r)
}
} else if r >= 'A' && r <= 'Z' {
lastHyphen = false
r += 'a' - 'A'
_, _ = b.WriteRune(r)
} else if (r == '_' || r == '/') && !lastHyphen { // Special-case for common characters that we still want to keep somehow
lastHyphen = true
_, _ = b.WriteRune('-')
}
// Else, skip it, silently.
if b.Len() == 40 {
// Technically, we could be less than 40 after some more cleaning... but this is good enough.
break
}
}
str := b.String()
// We must start and end with alphanumerics.
str = strings.Trim(str, "-")
str = strings.Trim(str, ".")
return str
}
// appendHashSuffix creates a function that modifies an input string to include a hash suffix.
// The goal of this function is to represent all needs for hashing IDs.
//
// The hash uses the input hashKey and repositoryName as a salt.
// The output string is at most 40 characters long. (253 is Kubernetes' limit, but UIDs have a max of 40 characters.)
// The output string is a valid Kubernetes name (see [sanitiseKubeName]).
// The output string contains at least 12 characters of a hash, plus a 1 character hyphen to separate the input and the hash.
// The function is deterministic given the same invocation parameters to this function.
func appendHashSuffix(hashKey, repositoryName string) func(string) string {
salt := []byte(repositoryName + "/" + hashKey)
const maxLen = 40 // valid Kubernetes name
const minSuffix = 8 // excluding hyphen
const minSpace = minSuffix + 1 // +1 for hyphen
return func(s string) string {
hasher := sha256.New()
// From hash.Hash docs:
// > Write (via the embedded io.Writer interface) adds more data to the running hash.
// > It never returns an error.
// As such, we ignore all errors.
_, _ = hasher.Write(salt) // Input to the parent function
_, _ = hasher.Write([]byte(s))
hash := base64.URLEncoding.EncodeToString(hasher.Sum(nil))
hash = sanitiseKubeName(hash) // We have rules to follow, as per our doc contract
if len(s) > maxLen-minSpace {
s = s[:maxLen-minSpace]
}
spaceForHash := maxLen - len(s) - 1
if spaceForHash < len(hash) {
hash = hash[:spaceForHash]
}
return sanitiseKubeName(s + "-" + hash)
}
}
// Will pick a name based on the hashed repository and path
func NamesFromHashedRepoPath(repo string, fpath string) (string, string) {
// Remove the extension: we don't want the extension to impact the ID. This lets the user change between all supported formats.
fpath = safepath.RemoveExt(fpath)
hasher := appendHashSuffix(fpath, repo)
return hasher(safepath.Base(fpath)), hasher(strings.Trim(safepath.Dir(fpath), "/"))
}
// Folder contains the data for a folder we use in provisioning.
type Folder struct {
// Title is the human-readable name created by a human who wrote it.
Title string
// ID represents the name the folder should have, derived from the title.
// It contains a suffix calculated from the path of the folder.
// The ID is used in Kubernetes and the folders API server. This is the same as the legacy (and by the time you read this, hopefully removed) UID concept of folders.
ID string
// Path is the full path to the folder, as given to the parse function.
Path string
}
func ParseFolder(dirPath, repositoryName string) Folder {
hasher := appendHashSuffix(strings.TrimSuffix(dirPath, "/"), repositoryName)
base := safepath.Base(dirPath)
return Folder{
Title: base,
ID: hasher(sanitiseKubeName(base)),
Path: dirPath,
}
}
func RootFolder(repository *provisioning.Repository) string {
if repository.Spec.Sync.Target == provisioning.SyncTargetTypeFolder {
return repository.Name // a folder with the same identifier as the repository
}
return ""
}
func ParentFolder(filePath string, repository *provisioning.Repository) string {
parent := safepath.Dir(filePath)
if parent == "" {
return RootFolder(repository)
}
return ParseFolder(parent, repository.GetName()).ID
}
pkg/registry/apis/provisioning/resources/id_test.go
+109
View File
@@ -0,0 +1,109 @@
package resources
import (
"strings"
"testing"
"github.com/stretchr/testify/assert"
)
func TestSanitiseKubeName(t *testing.T) {
for _, tc := range []struct{ Name, Input, Expected string }{
{"Valid Kubernetes name", "a-b.123.-c", "a-b.123.-c"},
{"Capital letters lowercase", "ABCDEFGHIJKLMNOPQRSTUVWXYZ", "abcdefghijklmnopqrstuvwxyz"},
{"Trailing hyphen removed", "abc-", "abc"},
{"Trailing dot removed", "abc.", "abc"},
{"Leading hyphen removed", "-abc", "abc"},
{"Leading dot removed", ".abc", "abc"},
{"Double hyphen simplified", "ab--c", "ab-c"},
{"Five hyphens simplified", "ab-----c", "ab-c"},
{"Underscore converted", "ab_c", "ab-c"},
{"Slash converted", "ab/c", "ab-c"},
{"Stops at 40 characters", strings.Repeat("a", 300), strings.Repeat("a", 40)},
} {
t.Run(tc.Name, func(t *testing.T) {
assert.Equal(t, tc.Expected, sanitiseKubeName(tc.Input))
})
}
}
func TestAppendHashSuffix(t *testing.T) {
t.Run("hash key is part of hash", func(t *testing.T) {
keyA := appendHashSuffix("A", "unit-test")("test")
keyB := appendHashSuffix("B", "unit-test")("test")
assert.NotEqual(t, keyA, keyB, "hash key is not part of hash")
})
t.Run("repository name is part of hash", func(t *testing.T) {
repoA := appendHashSuffix("unit-test", "A")("test")
repoB := appendHashSuffix("unit-test", "B")("test")
assert.NotEqual(t, repoA, repoB, "repo name is not part of hash")
})
t.Run("is deterministic", func(t *testing.T) {
hasher := appendHashSuffix("unit", "test")
assert.Equal(t, hasher("a"), hasher("a"))
_ = hasher("b") // Assume it isn't deterministic: this would likely modify a hasher state
assert.Equal(t, hasher("a"), hasher("a")) // alas, it is deterministic!
})
// These are covered by the tests above, so we can just use static values from now on
const repoName = "repository"
const hashKey = "key"
hasher := appendHashSuffix(hashKey, repoName)
for _, tc := range []struct{ Name, Input, Expected string }{
{"Simple, short prefix", "test", "test-ae2h65vh3ygoxmprmnludpyhhpr3d-5iosy"},
{"Suffix requiring cutting hash to min", strings.Repeat("test", 200), "testtesttesttesttesttesttesttes-8dogmh7b"},
{"Suffix requiring partially cutting hash", strings.Repeat("test", 7), "testtesttesttesttesttesttest-icauzj-i5j5"},
} {
t.Run(tc.Name, func(t *testing.T) {
hashed := hasher(tc.Input)
assert.Equal(t, tc.Expected, hashed, "hashed value must be as expected")
// These exist both because they're helpful to understand how something isn't equal above, and for programmer errors. (e.g. what if I manually input a 41 char expected value? or one with too few hash chars?)
// We only want 40 characters because UIDs support no more. When we get rid of legacy storage, we can extend the support to 253 character long strings.
assert.LessOrEqual(t, len(hashed), 40, "string after hashing needs to be <=40 chars long")
assert.GreaterOrEqual(t, len(strings.SplitAfterN(hashed, "-", 2)[1]), 8, "hash must be at least 8 characters long")
})
}
}
func TestNamesFromHashedRepoPath(t *testing.T) {
dashName, folderName := NamesFromHashedRepoPath("xyz", "path/to/folder/dashboard.json")
assert.Equal(t, "dashboard-fy2kflbmskvt6u-9uecoahd1ekwbb7", dashName, "dashboard name of dashboard.json")
assert.Equal(t, "path-to-folder-3ehfurpmbvs4yxfp7a0r2uskr", folderName, "folder name of dashboard.json")
// We only want 40 characters because UIDs support no more. When we get rid of legacy storage, we can extend the support to 253 character long strings.
assert.LessOrEqual(t, len(dashName), 40, "dashName after hashing needs to be <=40 chars long")
assert.LessOrEqual(t, len(folderName), 40, "folderName after hashing needs to be <=40 chars long")
name1, f1 := NamesFromHashedRepoPath("xyz", "path/to/folder.json")
name2, f2 := NamesFromHashedRepoPath("xyz", "path/to/folder")
assert.Equal(t, name1, name2, "folder.json should have same object name as folder (no extension)")
assert.Equal(t, f1, f2, "folder.json and folder should have same folder name as each other")
}
func TestParseFolderID(t *testing.T) {
const repoName = "unit-test" // we have other tests verifying the repo name changes the id
cases := []struct {
Description string
Path string
Title string
KubeName string
}{
{"Short, simple path", "hello/world", "world", "world-wik-hjayboohlsvzzr2ob3he8cs7ffk0jd"},
{"Capital letters and punctuation", "Hello, World!", "Hello, World!", "helloworld-sbcnvdmezf0jnvgfhpk5ewaoawbeg"},
{"Very long name", strings.Repeat("/hello/world", 200), "world", "world-bc9jpbg6ctg-w-pexkul-f1ic-bwer5-3r"},
}
for _, c := range cases {
t.Run(c.Description, func(t *testing.T) {
id := ParseFolder(c.Path, repoName)
assert.Equal(t, c.Path, id.Path)
assert.Equal(t, c.KubeName, id.ID)
assert.Equal(t, c.Title, id.Title)
// We only want 40 characters because UIDs support no more. When we get rid of legacy storage, we can extend the support to 253 character long strings.
assert.LessOrEqual(t, len(id.ID), 40, "ID after hashing needs to be <=40 chars long")
})
}
}
pkg/registry/apis/provisioning/resources/object.go
+165
View File
@@ -0,0 +1,165 @@
package resources
import (
"context"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/runtime/schema"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
"github.com/grafana/grafana/pkg/apimachinery/utils"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/dashboard/legacy"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
"github.com/grafana/grafana/pkg/storage/unified/resource"
)
// Get repository stats
type ResourceLister interface {
List(ctx context.Context, namespace, repository string) (*provisioning.ResourceList, error)
Stats(ctx context.Context, namespace, repository string) (*provisioning.ResourceStats, error)
}
type ResourceListerFromSearch struct {
managed resource.ManagedObjectIndexClient
index resource.ResourceIndexClient
legacyMigrator legacy.LegacyMigrator
storageStatus dualwrite.Service
}
func NewResourceLister(
managed resource.ManagedObjectIndexClient,
index resource.ResourceIndexClient,
legacyMigrator legacy.LegacyMigrator,
storageStatus dualwrite.Service,
) ResourceLister {
return &ResourceListerFromSearch{
index: index,
managed: managed,
legacyMigrator: legacyMigrator,
storageStatus: storageStatus,
}
}
// List implements ResourceLister.
func (o *ResourceListerFromSearch) List(ctx context.Context, namespace, repository string) (*provisioning.ResourceList, error) {
objects, err := o.managed.ListManagedObjects(ctx, &resource.ListManagedObjectsRequest{
Namespace: namespace,
Kind: string(utils.ManagerKindRepo),
Id: repository,
})
if err != nil {
return nil, err
}
if objects.Error != nil {
return nil, resource.GetError(objects.Error)
}
list := &provisioning.ResourceList{}
for _, v := range objects.Items {
list.Items = append(list.Items, provisioning.ResourceListItem{
Path: v.Path,
Group: v.Object.Group,
Resource: v.Object.Resource,
Name: v.Object.Name,
Hash: v.Hash,
Time: v.Time,
Title: v.Title,
Folder: v.Folder,
})
}
return list, nil
}
// Stats implements ResourceLister.
func (o *ResourceListerFromSearch) Stats(ctx context.Context, namespace, repository string) (*provisioning.ResourceStats, error) {
req := &resource.CountManagedObjectsRequest{
Namespace: namespace,
}
if repository != "" {
req.Kind = string(utils.ManagerKindRepo)
req.Id = repository
}
counts, err := o.managed.CountManagedObjects(ctx, req)
if err != nil {
return nil, err
}
if counts.Error != nil {
return nil, resource.GetError(counts.Error)
}
lookup := make(map[string]*provisioning.ManagerStats)
for _, v := range counts.Items {
key := v.Kind + ":" + v.Id
m := lookup[key]
if m == nil {
m = &provisioning.ManagerStats{
Kind: utils.ManagerKind(v.Kind),
Identity: v.Id,
}
lookup[key] = m
}
m.Stats = append(m.Stats, provisioning.ResourceCount{
Group: v.Group,
Resource: v.Resource,
Count: v.Count,
})
}
stats := &provisioning.ResourceStats{
TypeMeta: metav1.TypeMeta{
APIVersion: provisioning.SchemeGroupVersion.String(),
Kind: "ResourceStats",
},
}
for _, v := range lookup {
stats.Managed = append(stats.Managed, *v)
}
// When selecting an explicit repository, do not fetch global stats
if repository != "" {
return stats, nil
}
// Get the stats based on what a migration could support
if dualwrite.IsReadingLegacyDashboardsAndFolders(ctx, o.storageStatus) {
rsp, err := o.legacyMigrator.Migrate(ctx, legacy.MigrateOptions{
Namespace: namespace,
Resources: []schema.GroupResource{{
Group: dashboard.GROUP, Resource: dashboard.DASHBOARD_RESOURCE,
}, {
Group: folders.GROUP, Resource: folders.RESOURCE,
}},
WithHistory: false,
OnlyCount: true,
})
if err != nil {
return nil, err
}
for _, v := range rsp.Summary {
stats.Instance = append(stats.Instance, provisioning.ResourceCount{
Group: v.Group,
Resource: v.Resource,
Count: v.Count,
})
}
return stats, nil
}
// Get full instance stats
info, err := o.index.GetStats(ctx, &resource.ResourceStatsRequest{
Namespace: namespace,
})
if err != nil {
return nil, err
}
for _, v := range info.Stats {
stats.Instance = append(stats.Instance, provisioning.ResourceCount{
Group: v.Group,
Resource: v.Resource,
Count: v.Count,
})
}
return stats, nil
}
pkg/registry/apis/provisioning/resources/parser.go
+294
View File
@@ -0,0 +1,294 @@
package resources
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"path"
"gopkg.in/yaml.v3"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"k8s.io/apimachinery/pkg/runtime/schema"
"k8s.io/client-go/dynamic"
"github.com/grafana/grafana-app-sdk/logging"
dashboard "github.com/grafana/grafana/apps/dashboard/pkg/apis/dashboard/v0alpha1"
"github.com/grafana/grafana/pkg/apimachinery/apis/common/v0alpha1"
"github.com/grafana/grafana/pkg/apimachinery/utils"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/repository"
)
var (
ErrNamespaceMismatch = errors.New("the file namespace does not match target namespace")
)
type ParserFactory struct {
ClientFactory *ClientFactory
}
func (f *ParserFactory) GetParser(ctx context.Context, repo repository.Reader) (*Parser, error) {
config := repo.Config()
clients, err := f.ClientFactory.Clients(ctx, config.GetNamespace())
if err != nil {
return nil, err
}
urls, _ := repo.(repository.RepositoryWithURLs)
return &Parser{
repo: provisioning.ResourceRepositoryInfo{
Type: config.Spec.Type,
Title: config.Spec.Title,
Namespace: config.Namespace,
Name: config.Name,
},
urls: urls,
clients: clients,
}, nil
}
type Parser struct {
// The target repository
repo provisioning.ResourceRepositoryInfo
// for repositories that have URL support
urls repository.RepositoryWithURLs
// ResourceClients give access to k8s apis
clients *ResourceClients
}
type ParsedResource struct {
// Original file Info
Info *repository.FileInfo
// The repository details
Repo provisioning.ResourceRepositoryInfo
// Resource URLs
URLs *provisioning.ResourceURLs
// Check for classic file types (dashboard.json, etc)
Classic provisioning.ClassicFileType
// Parsed contents
Obj *unstructured.Unstructured
// Metadata accessor for the file object
Meta utils.GrafanaMetaAccessor
// The Kind is defined in the file
GVK *schema.GroupVersionKind
// The Resource is found by mapping Kind to the right apiserver
GVR *schema.GroupVersionResource
// Client that can talk to this resource
Client dynamic.ResourceInterface
// The Existing object (same name)
// ?? do we need/want the whole thing??
Existing *unstructured.Unstructured
// Create or Update
Action provisioning.ResourceAction
// The results from dry run
DryRunResponse *unstructured.Unstructured
// When the value has been saved in the grafana database
Upsert *unstructured.Unstructured
// If we got some Errors
Errors []error
}
func (r *Parser) Clients() *ResourceClients {
return r.clients
}
func (r *Parser) Parse(ctx context.Context, info *repository.FileInfo, validate bool) (parsed *ParsedResource, err error) {
logger := logging.FromContext(ctx).With("path", info.Path, "validate", validate)
parsed = &ParsedResource{
Info: info,
Repo: r.repo,
}
if err := IsPathSupported(info.Path); err != nil {
return parsed, err
}
if info.Path == "" {
return parsed, errors.New("path is required")
}
parsed.Obj, parsed.GVK, err = DecodeYAMLObject(bytes.NewBuffer(info.Data))
if err != nil {
logger.Debug("failed to find GVK of the input data", "error", err)
parsed.Obj, parsed.GVK, parsed.Classic, err = ReadClassicResource(ctx, info)
if err != nil {
logger.Debug("also failed to get GVK from fallback loader?", "error", err)
return parsed, err
}
}
if r.urls != nil {
parsed.URLs, err = r.urls.ResourceURLs(ctx, info)
if err != nil {
logger.Debug("failed to load resource URLs", "error", err)
return parsed, err
}
}
// Remove the internal dashboard UID,version and id if they exist
if parsed.GVK.Group == dashboard.GROUP && parsed.GVK.Kind == "Dashboard" {
unstructured.RemoveNestedField(parsed.Obj.Object, "spec", "uid")
unstructured.RemoveNestedField(parsed.Obj.Object, "spec", "version")
unstructured.RemoveNestedField(parsed.Obj.Object, "spec", "id") // now managed as a label
}
parsed.Meta, err = utils.MetaAccessor(parsed.Obj)
if err != nil {
return nil, err
}
obj := parsed.Obj
// Validate the namespace
if obj.GetNamespace() != "" && obj.GetNamespace() != r.repo.Namespace {
parsed.Errors = append(parsed.Errors, ErrNamespaceMismatch)
}
obj.SetNamespace(r.repo.Namespace)
parsed.Meta.SetManagerProperties(utils.ManagerProperties{
Kind: utils.ManagerKindRepo,
Identity: r.repo.Name,
})
parsed.Meta.SetSourceProperties(utils.SourceProperties{
Path: info.Path, // joinPathWithRef(info.Path, info.Ref),
Checksum: info.Hash,
})
// Calculate name+folder from the file path
if info.Path != "" {
objName, folderName := NamesFromHashedRepoPath(r.repo.Name, info.Path)
parsed.Meta.SetFolder(folderName)
if obj.GetName() == "" {
obj.SetName(objName) // use the name saved in config
}
}
obj.SetUID("") // clear identifiers
obj.SetResourceVersion("") // clear identifiers
// We can not do anything more if no kind is defined
if parsed.GVK == nil {
return parsed, nil
}
if r.clients == nil {
return parsed, fmt.Errorf("no client configured")
}
client, gvr, err := r.clients.ForKind(*parsed.GVK)
if err != nil {
return nil, err // does not map to a resour e
}
parsed.GVR = &gvr
parsed.Client = client
if !validate {
return parsed, nil
}
if parsed.Client == nil {
parsed.Errors = append(parsed.Errors, fmt.Errorf("unable to find client"))
return parsed, nil
}
// Dry run CREATE or UPDATE
parsed.Existing, _ = parsed.Client.Get(ctx, obj.GetName(), metav1.GetOptions{})
if parsed.Existing == nil {
parsed.Action = provisioning.ResourceActionCreate
parsed.DryRunResponse, err = parsed.Client.Create(ctx, obj, metav1.CreateOptions{
DryRun: []string{"All"},
})
} else {
parsed.Action = provisioning.ResourceActionUpdate
parsed.DryRunResponse, err = parsed.Client.Update(ctx, obj, metav1.UpdateOptions{
DryRun: []string{"All"},
})
}
if err != nil {
parsed.Errors = append(parsed.Errors, err)
}
return parsed, nil
}
func (f *ParsedResource) ToSaveBytes() ([]byte, error) {
// TODO? should we use the dryRun (validated) version?
obj := make(map[string]any)
for k, v := range f.Obj.Object {
if k != "metadata" {
obj[k] = v
}
}
switch path.Ext(f.Info.Path) {
// JSON pretty print
case ".json":
return json.MarshalIndent(obj, "", " ")
// Write the value as yaml
case ".yaml", ".yml":
return yaml.Marshal(obj)
default:
return nil, fmt.Errorf("unexpected format")
}
}
func (f *ParsedResource) AsResourceWrapper() *provisioning.ResourceWrapper {
info := f.Info
res := provisioning.ResourceObjects{
Type: provisioning.ResourceType{
Classic: f.Classic,
},
Action: f.Action,
}
if f.GVK != nil {
res.Type.Group = f.GVK.Group
res.Type.Version = f.GVK.Version
res.Type.Kind = f.GVK.Kind
}
// The resource (GVR) is derived from the kind (GVK)
if f.GVR != nil {
res.Type.Resource = f.GVR.Resource
}
if f.Obj != nil {
res.File = v0alpha1.Unstructured{Object: f.Obj.Object}
}
if f.Existing != nil {
res.Existing = v0alpha1.Unstructured{Object: f.Existing.Object}
}
if f.Upsert != nil {
res.Upsert = v0alpha1.Unstructured{Object: f.Upsert.Object}
} else if f.DryRunResponse != nil {
res.DryRun = v0alpha1.Unstructured{Object: f.DryRunResponse.Object}
}
wrap := &provisioning.ResourceWrapper{
Path: info.Path,
Ref: info.Ref,
Hash: info.Hash,
Repository: f.Repo,
URLs: f.URLs,
Timestamp: info.Modified,
Resource: res,
}
for _, err := range f.Errors {
wrap.Errors = append(wrap.Errors, err.Error())
}
return wrap
}
pkg/registry/apis/provisioning/resources/tree.go
+139
View File
@@ -0,0 +1,139 @@
package resources
import (
"context"
"fmt"
"sort"
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
"github.com/grafana/grafana/pkg/apimachinery/utils"
folders "github.com/grafana/grafana/pkg/apis/folder/v0alpha1"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/registry/apis/provisioning/safepath"
)
// FolderTree contains the entire set of folders (at a given snapshot in time) of the Grafana instance.
// The folders are portrayed as a tree, where a folder has a parent, up until the root folder.
// The root folder is special-cased as a folder that exists, but is not itself stored. It has no ID, no title, and no data, but will return `true` for OK bools.
type FolderTree struct {
tree map[string]string
folders map[string]Folder
}
// In determines if the given folder is in the tree at all. That is, it answers "does the folder even exist in the Grafana instance?"
// An empty folder string means the root folder, and is special-cased to always return true.
func (t *FolderTree) In(folder string) bool {
_, ok := t.tree[folder]
return ok || folder == ""
}
// DirPath creates the path to the directory with slashes, up to but not including the baseFolder.
// The baseFolder is expected to be the repository's root folder, as defined by its spec. If this is used in other contexts, it should still function.
// An empty folder string means the root folder, and is special-cased to return no ID data.
// The returned Path field in the ID is relative to the base folder. If it is the base folder, an empty string is returned.
//
// If In(folder) or In(baseFolder) is false, this will return ok=false, because it would be undefined behaviour.
// If baseFolder is not a parent of folder, ok=false is returned.
func (t *FolderTree) DirPath(folder, baseFolder string) (fid Folder, ok bool) {
if !t.In(folder) || !t.In(baseFolder) {
return Folder{}, false
}
if folder == "" && baseFolder != "" {
return Folder{}, false
} else if folder == baseFolder {
// Zero-value: we're fine with the zv if we're working with the root folder here.
// Any other folder ID will have the correct metadata and no path (which is correct).
return t.folders[folder], true
}
fid = t.folders[folder]
fid.Path = fid.Title
ok = baseFolder == ""
parent := t.tree[folder]
for parent != "" {
if parent == baseFolder {
ok = true
break
}
fid.Path = safepath.Join(t.folders[parent].Title, fid.Path)
parent = t.tree[parent]
}
return fid, ok
}
func (t *FolderTree) Add(folder Folder, parent string) {
t.tree[folder.ID] = parent
t.folders[folder.ID] = folder
}
type WalkFunc func(ctx context.Context, folder Folder) error
func (t *FolderTree) Walk(ctx context.Context, fn WalkFunc) error {
toWalk := make([]Folder, 0, len(t.folders))
for _, folder := range t.folders {
folder, _ := t.DirPath(folder.ID, "")
toWalk = append(toWalk, folder)
}
// sort by depth of the paths
sort.Slice(toWalk, func(i, j int) bool {
return safepath.Depth(toWalk[i].Path) < safepath.Depth(toWalk[j].Path)
})
for _, folder := range toWalk {
if err := fn(ctx, folder); err != nil {
return err
}
}
return nil
}
func NewEmptyFolderTree() *FolderTree {
return &FolderTree{
tree: make(map[string]string, 0),
folders: make(map[string]Folder, 0),
}
}
func (t *FolderTree) AddUnstructured(item *unstructured.Unstructured, skipRepo string) error {
meta, err := utils.MetaAccessor(item)
if err != nil {
return fmt.Errorf("extract meta accessor: %w", err)
}
manager, _ := meta.GetManagerProperties()
if manager.Identity == skipRepo {
return nil // skip it... already in tree?
}
folder := Folder{
Title: meta.FindTitle(item.GetName()),
ID: item.GetName(),
}
t.tree[folder.ID] = meta.GetFolder()
t.folders[folder.ID] = folder
return nil
}
func NewFolderTreeFromResourceList(resources *provisioning.ResourceList) *FolderTree {
tree := make(map[string]string, len(resources.Items))
folderIDs := make(map[string]Folder, len(resources.Items))
for _, rf := range resources.Items {
if rf.Group != folders.GROUP {
continue
}
tree[rf.Name] = rf.Folder
folderIDs[rf.Name] = Folder{
Title: rf.Title,
ID: rf.Name,
Path: rf.Path,
}
}
return &FolderTree{
tree,
folderIDs,
}
}
pkg/registry/apis/provisioning/resources/tree_test.go
+92
View File
@@ -0,0 +1,92 @@
package resources
import (
"testing"
"github.com/stretchr/testify/assert"
)
func TestFolderTree(t *testing.T) {
newFid := func(kube, title string) Folder {
return Folder{ID: kube, Title: title}
}
t.Run("empty tree", func(t *testing.T) {
tree := &FolderTree{
tree: make(map[string]string),
folders: make(map[string]Folder),
}
assert.False(t, tree.In("x"), "x should not be in tree")
assert.False(t, tree.In("z"), "z should not be in tree")
_, ok := tree.DirPath("x", "")
assert.False(t, ok, "x should not have a DirPath")
})
t.Run("single directory in tree", func(t *testing.T) {
tree := &FolderTree{
tree: map[string]string{"x": ""},
folders: map[string]Folder{"x": newFid("x", "X!")},
}
assert.True(t, tree.In("x"), "x should be in tree")
id, ok := tree.DirPath("x", "x")
if assert.True(t, ok, "x should have DirPath with itself as base") {
assert.Equal(t, "x", id.ID, "KubernetesName")
assert.Equal(t, "X!", id.Title, "Title")
assert.Equal(t, "", id.Path, "Path")
}
id, ok = tree.DirPath("x", "")
if assert.True(t, ok, "x should have DirPath with empty base") {
assert.Equal(t, "x", id.ID, "KubernetesName")
assert.Equal(t, "X!", id.Title, "Title")
assert.Equal(t, "X!", id.Path, "Path")
}
})
t.Run("simple nesting tree", func(t *testing.T) {
tree := &FolderTree{
tree: map[string]string{"a": "b", "b": "c", "c": "x", "x": ""},
folders: map[string]Folder{
"x": newFid("x", "X!"),
"c": newFid("c", "C :)"),
"b": newFid("b", "!!B#!"),
"a": newFid("a", "[€]@£a"),
},
}
assert.True(t, tree.In("x"), "x should be in tree")
assert.True(t, tree.In("a"), "a should be in tree")
assert.False(t, tree.In("z"), "z should not be in tree, for it is undeclared")
id, ok := tree.DirPath("x", "")
if assert.True(t, ok, "x should have DirPath with empty base") {
assert.Equal(t, "x", id.ID, "KubernetesName")
assert.Equal(t, "X!", id.Title, "Title")
assert.Equal(t, "X!", id.Path, "Path")
}
id, ok = tree.DirPath("c", "c")
if assert.True(t, ok, "c should have DirPath with itself as base") {
assert.Equal(t, "c", id.ID, "KubernetesName")
assert.Equal(t, "C :)", id.Title, "Title")
assert.Equal(t, "", id.Path, "Path")
}
id, ok = tree.DirPath("a", "x")
if assert.True(t, ok, "a should have DirPath with x as base") {
assert.Equal(t, "a", id.ID, "KubernetesName")
assert.Equal(t, "[€]@£a", id.Title, "Title")
assert.Equal(t, "C :)/!!B#!/[€]@£a", id.Path, "Path")
}
_, ok = tree.DirPath("x", "a")
assert.False(t, ok, "x should not have DirPath with a as base, because a is a subfolder of x")
id, ok = tree.DirPath("", "")
if assert.True(t, ok, "the root folder should have a path to itself") {
assert.Empty(t, id.ID)
assert.Empty(t, id.Path)
assert.Empty(t, id.Title)
}
})
}
pkg/registry/apis/provisioning/routes.go
+174
View File
@@ -0,0 +1,174 @@
package provisioning
import (
"encoding/json"
"fmt"
"net/http"
"time"
"k8s.io/apimachinery/pkg/labels"
"k8s.io/kube-openapi/pkg/spec3"
"k8s.io/kube-openapi/pkg/validation/spec"
authlib "github.com/grafana/authlib/types"
provisioning "github.com/grafana/grafana/pkg/apis/provisioning/v0alpha1"
"github.com/grafana/grafana/pkg/services/apiserver/builder"
"github.com/grafana/grafana/pkg/storage/legacysql/dualwrite"
"github.com/grafana/grafana/pkg/util/errhttp"
)
// TODO: Move the specific logic to the connector so that we don't have logic all over the place.
// GetAPIRoutes implements the direct HTTP handlers that bypass k8s
func (b *APIBuilder) GetAPIRoutes() *builder.APIRoutes {
return &builder.APIRoutes{
Namespace: []builder.APIRouteHandler{
{
Path: "stats",
Spec: &spec3.PathProps{
Get: &spec3.Operation{
OperationProps: spec3.OperationProps{
OperationId: "getResourceStats", // used for RTK client
Tags: []string{"Provisioning", "Repository"}, // includes stats for repositores and provisiong in general
Description: "Get resource stats for this namespace",
Parameters: []*spec3.Parameter{
{
ParameterProps: spec3.ParameterProps{
Name: "namespace",
In: "path",
Required: true,
Example: "default",
Description: "workspace",
Schema: spec.StringProperty(),
},
},
},
Responses: &spec3.Responses{
ResponsesProps: spec3.ResponsesProps{
StatusCodeResponses: map[int]*spec3.Response{
200: {
ResponseProps: spec3.ResponseProps{
Content: map[string]*spec3.MediaType{
"application/json": {
MediaTypeProps: spec3.MediaTypeProps{
Schema: &spec.Schema{
SchemaProps: spec.SchemaProps{
Ref: spec.MustCreateRef("#/components/schemas/com.github.grafana.grafana.pkg.apis.provisioning.v0alpha1.ResourceStats"),
},
},
},
},
},
},
},
},
},
},
},
},
},
Handler: withTimeoutFunc(b.handleStats, 30*time.Second),
},
{
Path: "settings",
Spec: &spec3.PathProps{
Get: &spec3.Operation{
OperationProps: spec3.OperationProps{
OperationId: "getFrontendSettings", // used for RTK client
// includes stats for repositores and provisiong in general
// This must include "Repository" so that the RTK client will invalidate when things are deleted
Tags: []string{"Provisioning", "Repository"},
Description: "Get the frontend settings for this namespace",
Parameters: []*spec3.Parameter{
{
ParameterProps: spec3.ParameterProps{
Name: "namespace",
In: "path",
Required: true,
Example: "default",
Description: "workspace",
Schema: spec.StringProperty(),
},
},
},
Responses: &spec3.Responses{
ResponsesProps: spec3.ResponsesProps{
StatusCodeResponses: map[int]*spec3.Response{
200: {
ResponseProps: spec3.ResponseProps{
Content: map[string]*spec3.MediaType{
"application/json": {
MediaTypeProps: spec3.MediaTypeProps{
Schema: &spec.Schema{
SchemaProps: spec.SchemaProps{
Ref: spec.MustCreateRef("#/components/schemas/com.github.grafana.grafana.pkg.apis.provisioning.v0alpha1.RepositoryViewList"),
},
},
},
},
},
},
},
},
},
},
},
},
},
Handler: withTimeoutFunc(b.handleSettings, 30*time.Second),
},
},
}
}
// TODO: why didn't we create a connector as we did before or have a separate file?
func (b *APIBuilder) handleStats(w http.ResponseWriter, r *http.Request) {
u, ok := authlib.AuthInfoFrom(r.Context())
if !ok {
w.WriteHeader(400)
_, _ = w.Write([]byte("expected user"))
return
}
// TODO: check if lister could list too many repositories or resources
stats, err := b.resourceLister.Stats(r.Context(), u.GetNamespace(), "")
if err != nil {
errhttp.Write(r.Context(), err, w)
return
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(stats)
}
// TODO: why didn't we create a connector as we did before or have a separate file?
// TODO: is there a better way to provide a filtered view of the repositories to the frontend?
func (b *APIBuilder) handleSettings(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
u, ok := authlib.AuthInfoFrom(ctx)
if !ok {
errhttp.Write(ctx, fmt.Errorf("expected user"), w)
return
}
// TODO: check if lister could list too many repositories or resources
all, err := b.repositoryLister.Repositories(u.GetNamespace()).List(labels.Everything())
if err != nil {
errhttp.Write(r.Context(), err, w)
return
}
settings := provisioning.RepositoryViewList{
Items: make([]provisioning.RepositoryView, len(all)),
// FIXME: this shouldn't be here in provisioning but at the dual writer or something about the storage
LegacyStorage: dualwrite.IsReadingLegacyDashboardsAndFolders(ctx, b.storageStatus),
}
for i, val := range all {
settings.Items[i] = provisioning.RepositoryView{
Name: val.ObjectMeta.Name,
Title: val.Spec.Title,
Type: val.Spec.Type,
ReadOnly: len(val.Spec.Workflows) == 0,
Target: val.Spec.Sync.Target,
}
}
w.Header().Set("Content-Type", "application/json")
_ = json.NewEncoder(w).Encode(settings)
}
pkg/registry/apis/provisioning/safepath/dir.go
+38
View File
@@ -0,0 +1,38 @@
package safepath
import (
"path"
"strings"
)
// IsDir returns true if the filePath ends with a slash.
// Empty string is considered a directory.
func IsDir(filePath string) bool {
if filePath == "" || filePath == "." {
return true
}
return strings.HasSuffix(filePath, "/")
}
// Dir behaves exactly as path.Dir, but returns "" for the root directory.
// and returns a trailing slash for all other directories.
func Dir(filePath string) string {
if filePath == "" {
return ""
}
// Trim trailing slash before getting the directory
cleanPath := strings.TrimSuffix(filePath, "/")
dir := path.Dir(cleanPath)
if dir == "." || dir == "/" {
return ""
}
return dir + "/"
}
// InDir returns true if the filePath is a subdirectory of the given directory.
func InDir(filePath, dir string) bool {
return strings.HasPrefix(filePath, dir)
}
pkg/registry/apis/provisioning/safepath/dir_test.go
+162
View File
@@ -0,0 +1,162 @@
package safepath
import (
"testing"
"github.com/stretchr/testify/require"
)
func TestIsFolderPath(t *testing.T) {
tests := []struct {
name string
filePath string
want bool
}{
{
name: "empty path",
filePath: "",
want: true,
},
{
name: "dot path",
filePath: ".",
want: true,
},
{
name: "file path without extension",
filePath: "test",
want: false,
},
{
name: "file path with extension",
filePath: "test.json",
want: false,
},
{
name: "folder path with trailing slash",
filePath: "folder/",
want: true,
},
{
name: "nested folder path with trailing slash",
filePath: "folder/subfolder/",
want: true,
},
{
name: "file path in folder without trailing slash",
filePath: "folder/test.json",
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := IsDir(tt.filePath)
require.Equal(t, tt.want, got)
})
}
}
func TestDir(t *testing.T) {
tests := []struct {
name string
filePath string
want string
}{
{
name: "empty path",
filePath: "",
want: "",
},
{
name: "root path",
filePath: "/",
want: "",
},
{
name: "single directory",
filePath: "folder",
want: "",
},
{
name: "nested directory",
filePath: "folder/subfolder",
want: "folder/",
},
{
name: "file in directory",
filePath: "folder/file.txt",
want: "folder/",
},
{
name: "multiple nested directories",
filePath: "folder/subfolder/subsubfolder",
want: "folder/subfolder/",
},
{
name: "directory with trailing slash",
filePath: "folder/subfolder/",
want: "folder/",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := Dir(tt.filePath)
require.Equal(t, tt.want, got)
})
}
}
func TestInDir(t *testing.T) {
tests := []struct {
name string
filePath string
dir string
want bool
}{
{
name: "file in directory",
filePath: "folder/file.txt",
dir: "folder/",
want: true,
},
{
name: "file not in directory",
filePath: "other/file.txt",
dir: "folder/",
want: false,
},
{
name: "subdirectory",
filePath: "folder/subfolder/",
dir: "folder/",
want: true,
},
{
name: "empty directory",
filePath: "folder/file.txt",
dir: "",
want: true,
},
{
name: "exact match",
filePath: "folder/",
dir: "folder/",
want: true,
},
{
name: "partial directory name match",
filePath: "folder2/file.txt",
dir: "folder/",
want: false,
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got := InDir(tt.filePath, tt.dir)
require.Equal(t, tt.want, got)
})
}
}
pkg/registry/apis/provisioning/safepath/path.go
+53 -40
View File
@@ -4,56 +4,69 @@ import (
"os"
"path"
"strings"
apierrors "k8s.io/apimachinery/pkg/api/errors"
)
// ErrUnsafePathTraversal indicates that an input path had a path traversal which led to escaping the required prefix.
// E.g. Join("/test", "..") would return this, because it doesn't stay within the '/test' directory.
var ErrUnsafePathTraversal = apierrors.NewBadRequest("the input path had an unacceptable path traversal")
// Join joins any number of elements in a path under a common prefix path.
// If the elems do path traversal, they are permitted to do so under their own directories.
// The output result will _always_ have a prefix of the given prefix, and no path traversals in the output string.
// The output result will not end with a trailing slash.
// The output result will have a leading slash if one is given as a prefix.
// If the prefix would ultimately be escaped, an error is returned.
//
// This function is safe for <https://securego.io/docs/rules/g304.html>.
func Join(prefix string, elem ...string) (string, error) {
// We clean early to make the HasPrefix check be sensible after path.Join does a Clean for us.
prefix = replaceOSSeparators(path.Clean(prefix))
if len(elem) == 0 {
return prefix, nil
}
for i, e := range elem {
// We don't use Clean here because the output of path.Join will clean for us.
elem[i] = replaceOSSeparators(e)
}
subPath := path.Join(elem...) // performs a Clean after joining
completePath := path.Join(prefix, subPath)
if !strings.HasPrefix(completePath, prefix) {
return "", ErrUnsafePathTraversal
}
return completePath, nil
}
// Performs a [path.Clean] on the path, as well as replacing its OS separators.
// Note that this does no effort to ensure the paths are safe to use. It only cleans them.
func Clean(p string) string {
return path.Clean(replaceOSSeparators(p))
}
// TODO: explore if we want to use our own type for safepath
// to make it clearer that this is a safe path and not a regular path
// osSeparator is declared as a var here only to ensure we can change it in tests.
var osSeparator = os.PathSeparator
// Performs a [path.Clean] on the path, as well as replacing its OS separators.
// This replaces the OS separator with a slash.
// All OSes we target (Linux, macOS, and Windows) support forward-slashes in path traversals, as such it's simpler to use the same character everywhere.
// BSDs do as well (even though they're not a target as of writing).
func replaceOSSeparators(p string) string {
func Clean(p string) string {
if osSeparator == '/' { // perf: nothing to do!
return path.Clean(p)
}
return path.Clean(strings.ReplaceAll(p, string(osSeparator), "/"))
}
// Join is like path.Join but preserves trailing slashes from the last element
func Join(elem ...string) string {
if len(elem) == 0 {
return ""
}
joined := path.Join(elem...)
// Preserve trailing slash if the last element had one
if strings.HasSuffix(elem[len(elem)-1], "/") {
return joined + "/"
}
return joined
}
// Base returns the last element of the path.
func Base(p string) string {
b := path.Base(p)
if b == "." || b == "/" {
return ""
}
return b
}
// RemoveExt returns the path without the extension.
// It should not remove the dot if the filename is e.g. `.gitignore`
func RemoveExt(p string) string {
// Special case: if the file starts with a dot and has no other dots,
// it's a hidden file and should not have its "extension" removed
base := Base(p)
if strings.HasPrefix(base, ".") && strings.Count(base, ".") == 1 {
return p
}
return strings.ReplaceAll(p, string(osSeparator), "/")
ext := path.Ext(p)
if ext == "" {
return p
}
return p[0 : len(p)-len(ext)]
}
func IsAbs(p string) bool {
return path.IsAbs(p)
}
pkg/registry/apis/provisioning/safepath/path_test.go
+114 -20
View File
@@ -7,41 +7,26 @@ import (
)
func TestPathJoin(t *testing.T) {
orig := osSeparator
osSeparator = '\\' // pretend we're on Windows
defer func() { osSeparator = orig }()
testCases := []struct {
Comment string
In []string
Out any // string or error
}{
{"Empty elements should not change input", []string{"/test/"}, "/test"},
{"Empty elements without leading slash should not change input", []string{"test/"}, "test"},
{"Empty elements should not change input", []string{"/test/"}, "/test/"},
{"Empty elements without leading slash should not change input", []string{"test/"}, "test/"},
{"Single element should be added to path", []string{"/test/", "abc"}, "/test/abc"},
{"Single element should be added to path with current dir prefix", []string{"./test/", "abc"}, "test/abc"},
{"Single element with leading slash should be added to path", []string{"/test/", "/abc"}, "/test/abc"},
{"Many elements are all appended to path", []string{"/test/", "a", "b", "c"}, "/test/a/b/c"},
{"Path traversal within same directory should be expanded", []string{"/test/", "a", "..", "b", ".", "..", "c"}, "/test/c"},
{"Path traversal escaping root dir prefix should return err", []string{"/test/", ".."}, ErrUnsafePathTraversal},
{"Path traversal escaping no dir prefix should return err", []string{"test/", ".."}, ErrUnsafePathTraversal},
{"Path traversal escaping current dir prefix should return err", []string{"./test/", ".."}, ErrUnsafePathTraversal},
{"Complex path traversal escaping prefix should return err", []string{"/test/", "a/..///c/", "../../test/d/../a/../.."}, ErrUnsafePathTraversal},
{"Complex path traversal remaining in prefix should be expanded", []string{"/test/", "a/..///c/", "../../test/d/"}, "/test/d"},
{"Problematic code example from the g304 website", []string{"/safe/path", "../../private/path"}, ErrUnsafePathTraversal},
{"Traversing beyond root should be expanded", []string{"/test/", "/../a"}, "/test/a"},
{"OS separator should be replaced with a slash", []string{"/test\\test", "abc\\test"}, "/test/test/abc/test"},
{"Complex path traversal remaining in prefix should be expanded", []string{"/test/", "a/..///c/", "../../test/d/"}, "/test/d/"},
}
for _, tc := range testCases {
tc := tc
t.Run(tc.Comment, func(t *testing.T) {
path, err := Join(tc.In[0], tc.In[1:]...)
if ee, ok := tc.Out.(error); ok {
assert.ErrorIs(t, err, ee, "expected unsuccessful outcome")
assert.Empty(t, path, "expected empty string when unsuccessful")
} else if str, ok := tc.Out.(string); ok {
assert.NoError(t, err, "expected successful outcome")
path := Join(tc.In...)
if str, ok := tc.Out.(string); ok {
assert.Equal(t, str, path)
} else {
panic("expected out was neither string nor error")
@@ -63,6 +48,13 @@ func TestPathClean(t *testing.T) {
{"Simple path", "/test/", "/test"},
{"Simple path with OS separators", "\\test\\here", "/test/here"},
{"Simple path with mixed separators", "\\test/here", "/test/here"},
{"Path traversal within directory", "/test/abc/../def", "/test/def"},
{"Multiple path traversals", "/test/abc/../../def", "/def"},
{"Path traversal beyond root", "/test/../../../def", "/def"},
{"Complex path traversal with mixed separators", "\\test\\abc\\..\\..\\def/ghi\\..", "/def"},
{"Path traversal with multiple slashes", "/test////abc/..//def", "/test/def"},
{"Path traversal with current directory", "/test/./abc/../def/./ghi", "/test/def/ghi"},
{"Empty path segments with traversal", "//test//abc//..//def", "/test/def"},
}
for _, tc := range testCases {
@@ -72,3 +64,105 @@ func TestPathClean(t *testing.T) {
})
}
}
func TestBase(t *testing.T) {
testCases := []struct {
name string
path string
expected string
}{
{
name: "empty path",
path: "",
expected: "",
},
{
name: "root path",
path: "/",
expected: "",
},
{
name: "current directory",
path: ".",
expected: "",
},
{
name: "simple filename",
path: "file.txt",
expected: "file.txt",
},
{
name: "path with directory",
path: "/path/to/file.txt",
expected: "file.txt",
},
{
name: "path with trailing slash",
path: "/path/to/dir/",
expected: "dir",
},
{
name: "hidden file",
path: ".gitignore",
expected: ".gitignore",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
result := Base(tc.path)
assert.Equal(t, tc.expected, result)
})
}
}
func TestRemoveExt(t *testing.T) {
testCases := []struct {
name string
path string
expected string
}{
{
name: "empty path",
path: "",
expected: "",
},
{
name: "no extension",
path: "filename",
expected: "filename",
},
{
name: "simple extension",
path: "file.txt",
expected: "file",
},
{
name: "multiple dots",
path: "file.tar.gz",
expected: "file.tar",
},
{
name: "hidden file",
path: ".gitignore",
expected: ".gitignore",
},
{
name: "path with directory",
path: "/path/to/file.txt",
expected: "/path/to/file",
},
{
name: "path with trailing slash",
path: "/path/to/dir/",
expected: "/path/to/dir/",
},
}
for _, tc := range testCases {
t.Run(tc.name, func(t *testing.T) {
result := RemoveExt(tc.path)
assert.Equal(t, tc.expected, result)
})
}
}

Some files were not shown because too many files have changed in this diff Show More