Commit Graph
3 Commits
Author SHA1 Message Date
763067f8e1 Dashboard Schema V2: Force v2 when dashboardNewLayouts or v2DashboardAPI are enabled (#113548)
* SchemaV2: Convertion from v1beta1 to v2beta1

* Compare backend-frontend v1 convertion

* Compare backend-frontend v1 convertion

* Fix fe be diff

* Resolve DS issues

* Fix ds inconsistecnies

* fix legacy string value issues

* fix ds test

* fix layout issue

* update test

* Fix tests and issue with defaultConfig

* Update output

* Fix viz config convertion

* wip

* Fix v1 to v2 dashboard transformation differences

Major fixes implemented:
- Backend function names in conversion.go
- Backend group field logic for queries, annotations, and vizConfig
- Backend datasource resolution with map-based lookup
- Backend timezone handling (empty string vs browser)
- Backend annotation processing (empty array vs default annotation)
- Backend default values (editable, liveNow)
- Backend variable processing (definition, defaultKeys, refresh, refId)
- Backend panel layout (y position calculations)
- Backend VizConfig (Kind and Group fields, default values)
- Frontend snapshot issue (annotations not processing)
- Frontend datasource references (only when original has valid datasource)

Test results:
- annotation-conversions: PASSING (0 differences)
- dashboard-properties: 3 expected architectural differences
- panel-conversions: Multiple expected architectural differences
- variable-conversions: 7 expected architectural differences

All remaining differences are expected architectural choices between
backend persistence optimization and frontend UI consumption optimization.

* fix issues with panel and annotation queries with no datasource

* definition and regex

* Use proper v1beta1 resource when testing

* remove misc file

* fix ds provider test

* fix def ds test in response transformer

* fix remaining ResponseTransformers test

* timesettings, variable refresh, editable, liveNow, definition

* fix transformSceneToSaveModelSchemaV2 test

* revert legacyRow changes

* fix go lint issues

* normalize y coordinates when serializing a row

* clean up

* update tests

* use GetStringValue from schemaversion

* fix go lint - cyclomatic complexity

* update open api snapshot

* add migrated dashboards

* fix default panel type when panel type is not provided

* revert dash link changes for now

* fix

* fix nested panel issue and default ref in v1

* apply defaults to nested panels too

* update snapshots

* fix issues with annotations

* matchers, showLegend, annotations

* when converting also don't process queries that have only a refId

* fix issues with text var

* fix dash links

* default to collapse: false when serializing

* fix: filter refId from variable query specs in backend migration

- Add buildDataQueryKindForVariable function to filter refId for variables
- Remove default refId "A" in transformSingleQuery
- Only include __legacyStringValue for non-empty string queries
- Remove refId addition in transformSaveModelSchemaV2ToScene.getDataQueryForVariable
- Handle undefined queries gracefully in frontend and backend
- Ensure backend matches frontend behavior for query variable serialization

* fix: default variable refresh to 'never' to match frontend behavior

Change backend default for missing refresh field from 'onDashboardLoad'
to 'never' to match frontend defaultVariableRefresh() schema default

* fix: only include iconColor in annotations when it exists

- Frontend: Use defaultAnnotationQuerySpec().iconColor as fallback to match schema defaults
- Backend: Only set iconColor if it exists in v1 input (not using GetStringValue)
- Ensures iconColor is only included when present in original dashboard

* fix: use schema defaults for annotation enable, hide, and iconColor

- Use defaultAnnotationQuerySpec() to get schema defaults instead of hardcoded values
- Default enable to false (schema default) to match frontend behavior
- Use schema default for iconColor and hide fields
- Ensures consistency with frontend which uses defaultAnnotationQuerySpec() defaults

* fix: set collapse for hidden-header rows to match first explicit row

- When panels appear before the first explicit row, the hidden-header row's
  collapse should match the first explicit row's collapsed value
- Matches frontend behavior where collapse: panel.collapsed uses the next
  row panel's collapsed value
- Ensures consistency between frontend and backend when converting rows layout

* fix: handle constant variables with missing query value

- Frontend: Fix bug where undefined value was converted to string 'undefined'
  - Now defaults to empty string when value is undefined: value ? String(value) : ''
- Backend: Match frontend fix - default to empty string for text/value when query is missing
- Ensures consistency when constant variable query is missing from v1 dashboard

* Fix interval variable handling when query is missing

- Extract intervals from options when query is missing/empty (matches backend behavior)
- Handle undefined/null query in getIntervalsFromQueryString
- Handle missing current object/value in getCurrentValueForOldIntervalModel
- Update interval variable refresh to use literal 'onTimeRangeChanged' in schema
- Use defaultIntervalVariableSpec() for interval variable serialization
- Backend: Generate query string from options when query is missing

* Fix corrupted dashboard with systemRef override

* don't resolve types for template variables in datasource refs on the backend

* fix annotation and ds issues

* fix range and special mappings

* fix datasource var pluginId and regex

* add __systemRef to schema

* update v15 migration annotation to have a ds type because v2 keeps track of if type is in the initial save model, and if it's not it removes it, but for frontendOuput we are running transformSaveModelToScene which will then assign the type

* add migration fields since the backend applies automigrations in collapsed rows

* filter out queries in ResponseTransformer that only have refId field

* lint

* v2: add default query if queries are empty to match v1 behavior

* fix single migration test

* tracking test should have a defined spec otherwise datasource is removed and won't be tracked

* initialize default with default ds ref

* wip

* Do not assign DS if ds group is empty

* cleanup

* revert change in setupTests.ts

* clean up TODO

* query with only refId should not expect to have a group

* refactor: extract v0alpha1 to v1beta1 conversion logic into atomic function

- Extract ConvertDashboard_V0_to_V1beta1 into v0alpha1_to_v1beta1.go
- Extract prepareV0ConversionContext and migrateV0Dashboard helper functions
- Standardize v0.go to match v1.go pattern with inline multi-step conversions
- Implement Convert_V0_to_V2alpha1 using atomic functions (v0->v1beta1->v2alpha1)
- Implement Convert_V0_to_V2beta1 using atomic functions (v0->v1beta1->v2alpha1->v2beta1)
- Remove non-atomic v0alpha1_to_v2alpha1.go file

* test: add version-specific test files for conversion error handling

- Extract v0 conversion tests into v0_test.go
- Extract v1 conversion tests into v1_test.go
- Add v2 conversion tests in v2_test.go
- Ensure all error handling paths in conversion functions are covered
- Add tests for Convert_V0_to_V2alpha1 and Convert_V0_to_V2beta1 error paths
- Add tests for Convert_V1beta1_to_V2alpha1 and Convert_V1beta1_to_V2beta1 error paths
- Add tests for Convert_V2alpha1_to_V2beta1 error handling

* Fix tests

* Fix linter

* Clean up

* feat(dashboard): Add automatic data loss detection for dashboard conversions

Implements comprehensive data loss detection for all dashboard API version conversions.

Components Tracked:
• Panels (visualization + library panels)
• Queries (data source queries, excludes row panel queries)
• Annotations
• Links
• Variables (template variables)

Features:
• Automatic detection via withConversionMetrics wrapper (zero code changes)
• Error type: 'conversion_data_loss_error'
• Logs: panelsLost, queriesLost, annotationsLost, linksLost, variablesLost

Bugs Found:
• Fixed critical bug: metrics.go was silently swallowing ALL errors (return nil → return err)

Testing:
• TestDataLossDetectionOnAllInputFiles - runs all conversions with detailed logging
• V2→V0/V1 downgrades write output for debugging then skip (not yet implemented)
• All tests passing

* Run dashboards on schema v2 E2Es

* reveret unintended changes

* cleanup

* Reset active manager correctly according to toggles config

* Fix new dashboard being serialized as v1

* Rename toggle

---------

Co-authored-by: Ivan Ortega <ivanortegaalba@gmail.com>
Co-authored-by: Dominik Prokop <dominik.prokop@grafana.com>
2025-11-27 15:52:42 +01:00
Ashley Harrison 6408e3acaa E2E: Remove cypress dashboards-suite (#109038)
* add equivalent dashboard-time-zone test

* remove cypress dashboards-suite

* modify tests to work with schema-v2 + update workflow to run playwright instead

* fix package.json

* update CODEOWNERS

* fix start-server to include ARCH
2025-08-05 10:09:49 +01:00
b6580ccb10 E2E: migrate to playwright (#107241)
* separate playwright into its own folder

* better separation

* add login scenario, add tags

* remove ui option

* update CODEOWNERS

* add a panels suite test

* migrate queries test to playwright

* rename + add dashlist test

* add panelEdit_base

* add geomap map controls test

* add geomap-layer-types

* add geomap-spatial-operations tests

* add frontend-sandbox-panel tests

* add smoke-tests-suite

* add comment about adding datasource

* add dashboard-browse-nested

* add dashboard-browse

* add dashboard-export-json

* add dashboard-keybindings test

* remove @wip tag

* turn on screenshots and add comment for why this test fails

* add dashboard-links-without-slug test

* try adding permissions in the test as well

* add dashboard-live-streaming

* context in the test doesn't work - sad

* create dashboard-public-templating

* add dashboard-public-create and make live streaming more resilient

* add share externally test

* add dashboard-share-internally

* add share-snapshot-create test

* add dashboard-templating

* add timepicker tests

* add embedded-dashboard test

* add general_dashboards test

* add import-dashboard test

* add load-options-from-url test

* add new-constant-variable test

* add custom-variable test

* add new-datasource-variable test

* add new-interval-variable test

* add text-box-variable test

* add new-query-variable test

* add horizontal repeat test

* add panel-vertical-repeat test

* add empty-row-repeat test

* add set-options-from-ui test

* add snapshot-create test

* add templating test

* add textbox-variables test

* add cloud-plugins-suite

* add storybook verification tests

* add playwright storybook verification workflow

* add playwright browsers

* update CODEOWNERS

* test change to trigger storybook verification workflows

* try container instead

* get the version right...

* go back to installing - less chance of forgetting to update

* Basic Github Actions

Squashed commit of the following:

commit f84c650a71
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 13:23:46 2025 +0100

    add arg for sharding, but not using it yet

commit 7bcf0512c6
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 12:30:30 2025 +0100

    less newline

commit b643911882
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 12:24:31 2025 +0100

    less logs

commit 38f871e9c2
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 10:00:26 2025 +0100

    fix yaml

commit db9a773136
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 09:57:47 2025 +0100

    clean up files

commit c0525f41fa
Author: joshhunt <josh.hunt@grafana.com>
Date:   Tue Jul 1 09:44:56 2025 +0100

    gha workflow

commit 895bea7c52
Author: joshhunt <josh.hunt@grafana.com>
Date:   Mon Jun 30 19:33:08 2025 +0100

    working dagger

commit cea1f84437
Author: joshhunt <josh.hunt@grafana.com>
Date:   Mon Jun 30 16:17:46 2025 +0100

    wip

* shard gha

* some tidy up

* add flags for exporting results, and a gha step to merge runs

* fix shard gha

* add dashboard-duplicate-panel test

* add dashboard-outline test

* add dashboards-add-panel

* remove some commented out code

* add dashboards-title-description test

* add dashboards-remove-panel

* don't install cypress

* gha: check playwright results

* add dashboards-edit-adhoc-variables test

* fix check-jobs

* add dagger cloud token

* add dagger cloud token

* add edit-datasource-variable test

* update CODEOWNERS

* add dashboards-edit-group-by-variables (skipped for now)

* add dashboards-edit-panel-title-description test

* add dashboards-edit-transparent-bg test

* add dashboards-edit-query-variables test

* run with 8 shards

* add dashboards-edit-variables

* tidy up gha

* add dashboard-group-panels

* fix action

* try to cache the grafana build

* fix missing action becuase no checkout, use builtin continue-on-error instead

* fix missing id

* cat out.txt

* debug build cache

* fix debug build cache

* add dashboards-panel-layouts test

* tidy up

* no more debug

* fix grafana dir

* add dashboards-move-panel test

* skip some failing tests

* mark up plugins tests with @plugins tag, only run @plugins tests in drone

* Hackathon/Playwright Conversion - Various Suite (#107516)

* Playwright Migration: Various Suite tests

* skipping bad tests

* fix some tests that can fail

* fix uid

* separate user for the verify-i18n test

* build test plugins for grafana server

* properly blur input fields

* login manually

* get dashboardPage from goto

* ignore a couple of type assertions

* remove a couple of timeouts

* remove timeouts on dashboard-share-internally

* use toBeHidden

* make dashboard-share-internally more stable

* remove TEMP_DAGGER_TOKEN

* clean up visaulization-suggestions

* unskip gauge test

* unskip trace-view-scrolling

* attempt to make create variable utils stable

* unskip loki tests

* make go linter happy

* unskip edit-group-by-variables test

* unskip move panel tests

* isolate dashboard-timepicker tests with separate user

* create data source as part of smoke test

* make sure we're awaiting in dashboard-edit-adhoc-variables

* make dashboards-edit-variables test more robust

* Hackathon Playwright: Dashboards Search (#107580)

* Hackathon Playwright: Dashboards Search

* Feedback changes

* make trace-view-scrolling more stable

* add json report and bench step

* fix bench version

* move fail step to after the playwright report so we can report test failures

* fix output file name

* fix typo

* try wrap in expect.poll

* stability

* bit more tidy up

* fix dashboard-new-layouts tests

* move test-plugins to e2e-playwright

* fix go code for drone e2e run

* move loki plugin-e2e test

* make v2 dashboards work again

---------

Co-authored-by: joshhunt <josh.hunt@grafana.com>
Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com>
Co-authored-by: Collin Fingar <collin.fingar@grafana.com>
Co-authored-by: Jeff Levin <jeff@levinology.com>
2025-07-11 10:31:33 +01:00