E2E: migrate to playwright (#107241)
* separate playwright into its own folder * better separation * add login scenario, add tags * remove ui option * update CODEOWNERS * add a panels suite test * migrate queries test to playwright * rename + add dashlist test * add panelEdit_base * add geomap map controls test * add geomap-layer-types * add geomap-spatial-operations tests * add frontend-sandbox-panel tests * add smoke-tests-suite * add comment about adding datasource * add dashboard-browse-nested * add dashboard-browse * add dashboard-export-json * add dashboard-keybindings test * remove @wip tag * turn on screenshots and add comment for why this test fails * add dashboard-links-without-slug test * try adding permissions in the test as well * add dashboard-live-streaming * context in the test doesn't work - sad * create dashboard-public-templating * add dashboard-public-create and make live streaming more resilient * add share externally test * add dashboard-share-internally * add share-snapshot-create test * add dashboard-templating * add timepicker tests * add embedded-dashboard test * add general_dashboards test * add import-dashboard test * add load-options-from-url test * add new-constant-variable test * add custom-variable test * add new-datasource-variable test * add new-interval-variable test * add text-box-variable test * add new-query-variable test * add horizontal repeat test * add panel-vertical-repeat test * add empty-row-repeat test * add set-options-from-ui test * add snapshot-create test * add templating test * add textbox-variables test * add cloud-plugins-suite * add storybook verification tests * add playwright storybook verification workflow * add playwright browsers * update CODEOWNERS * test change to trigger storybook verification workflows * try container instead * get the version right... * go back to installing - less chance of forgetting to update * Basic Github Actions Squashed commit of the following: commitf84c650a71Author: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 13:23:46 2025 +0100 add arg for sharding, but not using it yet commit7bcf0512c6Author: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 12:30:30 2025 +0100 less newline commitb643911882Author: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 12:24:31 2025 +0100 less logs commit38f871e9c2Author: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 10:00:26 2025 +0100 fix yaml commitdb9a773136Author: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 09:57:47 2025 +0100 clean up files commitc0525f41faAuthor: joshhunt <josh.hunt@grafana.com> Date: Tue Jul 1 09:44:56 2025 +0100 gha workflow commit895bea7c52Author: joshhunt <josh.hunt@grafana.com> Date: Mon Jun 30 19:33:08 2025 +0100 working dagger commitcea1f84437Author: joshhunt <josh.hunt@grafana.com> Date: Mon Jun 30 16:17:46 2025 +0100 wip * shard gha * some tidy up * add flags for exporting results, and a gha step to merge runs * fix shard gha * add dashboard-duplicate-panel test * add dashboard-outline test * add dashboards-add-panel * remove some commented out code * add dashboards-title-description test * add dashboards-remove-panel * don't install cypress * gha: check playwright results * add dashboards-edit-adhoc-variables test * fix check-jobs * add dagger cloud token * add dagger cloud token * add edit-datasource-variable test * update CODEOWNERS * add dashboards-edit-group-by-variables (skipped for now) * add dashboards-edit-panel-title-description test * add dashboards-edit-transparent-bg test * add dashboards-edit-query-variables test * run with 8 shards * add dashboards-edit-variables * tidy up gha * add dashboard-group-panels * fix action * try to cache the grafana build * fix missing action becuase no checkout, use builtin continue-on-error instead * fix missing id * cat out.txt * debug build cache * fix debug build cache * add dashboards-panel-layouts test * tidy up * no more debug * fix grafana dir * add dashboards-move-panel test * skip some failing tests * mark up plugins tests with @plugins tag, only run @plugins tests in drone * Hackathon/Playwright Conversion - Various Suite (#107516) * Playwright Migration: Various Suite tests * skipping bad tests * fix some tests that can fail * fix uid * separate user for the verify-i18n test * build test plugins for grafana server * properly blur input fields * login manually * get dashboardPage from goto * ignore a couple of type assertions * remove a couple of timeouts * remove timeouts on dashboard-share-internally * use toBeHidden * make dashboard-share-internally more stable * remove TEMP_DAGGER_TOKEN * clean up visaulization-suggestions * unskip gauge test * unskip trace-view-scrolling * attempt to make create variable utils stable * unskip loki tests * make go linter happy * unskip edit-group-by-variables test * unskip move panel tests * isolate dashboard-timepicker tests with separate user * create data source as part of smoke test * make sure we're awaiting in dashboard-edit-adhoc-variables * make dashboards-edit-variables test more robust * Hackathon Playwright: Dashboards Search (#107580) * Hackathon Playwright: Dashboards Search * Feedback changes * make trace-view-scrolling more stable * add json report and bench step * fix bench version * move fail step to after the playwright report so we can report test failures * fix output file name * fix typo * try wrap in expect.poll * stability * bit more tidy up * fix dashboard-new-layouts tests * move test-plugins to e2e-playwright * fix go code for drone e2e run * move loki plugin-e2e test * make v2 dashboards work again --------- Co-authored-by: joshhunt <josh.hunt@grafana.com> Co-authored-by: Josh Hunt <joshhunt@users.noreply.github.com> Co-authored-by: Collin Fingar <collin.fingar@grafana.com> Co-authored-by: Jeff Levin <jeff@levinology.com>
This commit is contained in:
Ashley Harrison
GitHub
@@ -7,12 +7,17 @@ on:
|
||||
- main
|
||||
- release-*.*.*
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
|
||||
# TODO: re-enable this before merging
|
||||
# concurrency:
|
||||
# group: ${{ github.workflow }}-${{ github.ref }}
|
||||
# cancel-in-progress: ${{ startsWith(github.ref, 'refs/pull/') }}
|
||||
|
||||
permissions: {}
|
||||
|
||||
env:
|
||||
ACTIONS_STEP_DEBUG: true
|
||||
RUNNER_DEBUG: 1
|
||||
|
||||
jobs:
|
||||
detect-changes:
|
||||
name: Detect whether code changed
|
||||
@@ -44,22 +49,56 @@ jobs:
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
path: ./grafana
|
||||
persist-credentials: false
|
||||
- uses: dagger/dagger-for-github@e47aba410ef9bb9ed81a4d2a97df31061e5e842e
|
||||
|
||||
# TODO: add a cleanup workflow to remove the cache when the PR is closed
|
||||
# https://github.com/actions/cache/blob/main/tips-and-workarounds.md#force-deletion-of-caches-overriding-default-cache-eviction-policy
|
||||
# TODO: maybe we could just use the cache to store the build, instead of uploading as an artifact?
|
||||
- uses: actions/cache@v4
|
||||
id: cache
|
||||
with:
|
||||
key: "build-grafana-${{ runner.os }}-${{ hashFiles('yarn.lock', 'public/*', 'packages/*', 'pkg/**/*.go', '**/go.mod', '**/go.sum', '!**_test.go', '!**.test.ts', '!**.test.tsx') }}"
|
||||
path: |
|
||||
build-dir
|
||||
|
||||
# If no cache hit, build Grafana
|
||||
- name: Build Grafana
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
uses: dagger/dagger-for-github@e47aba410ef9bb9ed81a4d2a97df31061e5e842e
|
||||
with:
|
||||
verb: run
|
||||
args: go -C grafana run ./pkg/build/cmd artifacts -a targz:grafana:linux/amd64 --grafana-dir="${PWD}/grafana" > out.txt
|
||||
- run: mv "$(cat out.txt)" grafana.tar.gz
|
||||
- run: echo "artifact=grafana-e2e-${{github.run_number}}" >> "$GITHUB_OUTPUT"
|
||||
args: go run ./pkg/build/cmd artifacts -a targz:grafana:linux/amd64 --grafana-dir="${PWD}" > out.txt
|
||||
- name: Cat built artifact
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
run: cat out.txt
|
||||
- name: Move built artifact
|
||||
if: steps.cache.outputs.cache-hit != 'true'
|
||||
run: |
|
||||
mkdir -p build-dir
|
||||
mv "$(cat out.txt)" build-dir/grafana.tar.gz
|
||||
|
||||
# If cache hit, validate the artifact is present
|
||||
- name: Validate artifact
|
||||
if: steps.cache.outputs.cache-hit == 'true'
|
||||
run: |
|
||||
if [ ! -f build-dir/grafana.tar.gz ]; then
|
||||
echo "Error: build-dir/grafana.tar.gz not found in cache"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Set artifact name
|
||||
run: echo "artifact=grafana-server-${{github.run_number}}" >> "$GITHUB_OUTPUT"
|
||||
id: artifact
|
||||
- uses: actions/upload-artifact@v4
|
||||
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-artifact@v4
|
||||
id: upload
|
||||
with:
|
||||
retention-days: 1
|
||||
name: ${{ steps.artifact.outputs.artifact }}
|
||||
path: grafana.tar.gz
|
||||
path: build-dir/grafana.tar.gz
|
||||
|
||||
# TODO: we won't need this when we only have playwright
|
||||
build-e2e-runner:
|
||||
needs: detect-changes
|
||||
if: needs.detect-changes.outputs.changed == 'true'
|
||||
@@ -159,6 +198,119 @@ jobs:
|
||||
path: videos
|
||||
retention-days: 1
|
||||
|
||||
run-playwright-tests:
|
||||
needs:
|
||||
- build-grafana
|
||||
name: Playwright E2E tests (${{ matrix.shard }}/${{ matrix.shardTotal }})
|
||||
runs-on: ubuntu-latest-8-cores
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
shard: [1, 2, 3, 4, 5, 6, 7, 8]
|
||||
shardTotal: [8]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
- uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: ${{ needs.build-grafana.outputs.artifact }}
|
||||
- name: Run E2E tests
|
||||
uses: dagger/dagger-for-github@e47aba410ef9bb9ed81a4d2a97df31061e5e842e
|
||||
with:
|
||||
verb: run
|
||||
args: go run ./pkg/build/e2e-playwright --package=grafana.tar.gz --shard=${{ matrix.shard }}/${{ matrix.shardTotal }} --blob-dir=./blob-report
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: success() || failure()
|
||||
with:
|
||||
name: playwright-blob-${{ github.run_number }}-${{ matrix.shard }}
|
||||
path: ./blob-report
|
||||
retention-days: 1
|
||||
|
||||
required-playwright-tests:
|
||||
needs:
|
||||
- run-playwright-tests
|
||||
- build-grafana
|
||||
if: ${{ !cancelled() }}
|
||||
name: All Playwright tests complete
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version-file: '.nvmrc'
|
||||
|
||||
- name: Download blob reports from GitHub Actions Artifacts
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
path: blobs
|
||||
pattern: playwright-blob-*
|
||||
merge-multiple: true
|
||||
|
||||
- name: Check blob reports
|
||||
run: |
|
||||
if [ ! "$(ls -A ./blobs)" ]; then
|
||||
echo "Error: No blob reports found in ./blobs directory"
|
||||
echo "Did the Playwright tests run at all?"
|
||||
exit 1
|
||||
fi
|
||||
echo "Found blob reports in ./blobs:"
|
||||
ls -lah ./blobs
|
||||
|
||||
- name: Merge into HTML Report
|
||||
run: npx playwright merge-reports --reporter html ./blobs
|
||||
|
||||
- name: Merge into JSON Report
|
||||
env:
|
||||
PLAYWRIGHT_JSON_OUTPUT_NAME: /tmp/playwright-results.json
|
||||
run: npx playwright merge-reports --reporter=json ./blobs
|
||||
|
||||
- name: Bench report
|
||||
run: |
|
||||
docker run --rm \
|
||||
--volume="/tmp/playwright-results.json:/home/bench/tests/playwright-results.json" \
|
||||
us-docker.pkg.dev/grafanalabs-global/docker-grafana-bench-prod/grafana-bench:v0.5.1 report \
|
||||
--grafana-url "http://localhost:3000" \
|
||||
--grafana-version "CI- ${{ github.sha }}" \
|
||||
--test-suite-name "FrontendCore" \
|
||||
--report-input playwright \
|
||||
--report-output log \
|
||||
--log-level DEBUG \
|
||||
/home/bench/tests/playwright-results.json
|
||||
|
||||
- name: Upload HTML report
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: playwright-html-${{ github.run_number }}
|
||||
path: playwright-report
|
||||
retention-days: 7
|
||||
|
||||
- name: Check test suites
|
||||
id: check-jobs
|
||||
uses: ./.github/actions/check-jobs
|
||||
continue-on-error: true # Failure will be reported on Show test results step
|
||||
with:
|
||||
needs: ${{ toJson(needs) }}
|
||||
failure-message: "One or more E2E test suites have failed"
|
||||
success-message: "All E2E test suites completed successfully"
|
||||
|
||||
- name: Show test results
|
||||
env:
|
||||
FAILED: ${{ steps.check-jobs.outputs.any-failed }}
|
||||
# sed removes the leading `../../src/` from the paths
|
||||
run: |
|
||||
npx playwright merge-reports --reporter list ./all-blob-reports | sed 's|\(\.\./\)\{1,\}src/|/|g'
|
||||
if [ "$FAILED" = "true" ]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
run-a11y-test:
|
||||
needs:
|
||||
- build-grafana
|
||||
@@ -193,6 +345,7 @@ jobs:
|
||||
required-e2e-tests:
|
||||
needs:
|
||||
- run-e2e-tests
|
||||
- build-grafana
|
||||
# a11y test is not listed on purpose: it is not an important E2E test.
|
||||
# It is also totally fine to fail right now.
|
||||
# always() is the best function here.
|
||||
@@ -203,13 +356,13 @@ jobs:
|
||||
name: All E2E tests complete
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Check test suites
|
||||
env:
|
||||
NEEDS: ${{ toJson(needs) }}
|
||||
run: |
|
||||
FAILURES="$(echo "$NEEDS" | jq 'with_entries(select(.value.result == "failure")) | map_values(.result)')"
|
||||
echo "$FAILURES"
|
||||
if [ "$(echo "$FAILURES" | jq '. | length')" != "0" ]; then
|
||||
exit 1
|
||||
fi
|
||||
echo "All OK!"
|
||||
uses: ./.github/actions/check-jobs
|
||||
with:
|
||||
needs: ${{ toJson(needs) }}
|
||||
failure-message: "One or more E2E test suites have failed"
|
||||
success-message: "All E2E test suites completed successfully"
|
||||
|
||||
Reference in New Issue
Block a user