diff --git a/.circleci/config.yml b/.circleci/config.yml index 6e42a7c7c8c..03a79831ce8 100644 --- a/.circleci/config.yml +++ b/.circleci/config.yml @@ -524,11 +524,8 @@ jobs: command: ./e2e/run-suite no_output_timeout: 5m - store_artifacts: - path: e2e/suite1/screenshots/expected - destination: expected-screenshots - - store_artifacts: - path: e2e/suite1/screenshots/received - destination: received-screenshots + path: e2e/suite1/screenshots + destination: screenshots - store_artifacts: path: e2e/suite1/videos destination: output-videos diff --git a/.gitignore b/.gitignore index cc6bb1ee8ff..a808efad312 100644 --- a/.gitignore +++ b/.gitignore @@ -105,5 +105,6 @@ compilation-stats.json /packages/grafana-e2e/cypress/videos /packages/grafana-e2e/cypress/logs /e2e/server.log -/e2e/suite1/screenshots/received +/e2e/suite1/screenshots +!/e2e/suite1/screenshots/expeced/* /e2e/suite1/videos/* diff --git a/contribute/style-guides/e2e.md b/contribute/style-guides/e2e.md index 434bf394eac..005097c477f 100644 --- a/contribute/style-guides/e2e.md +++ b/contribute/style-guides/e2e.md @@ -1,19 +1,43 @@ # End to end test framework + Grafana Labs uses a minimal home grown solution built on top of Cypress for our end to end (e2e) tests. +## Commands + +- `yarn e2e` Creates an isolated grafana-server home under `/e2e/tmp` with provisioned data sources and dashboards. This + copies locally build binary and frontned assets from your repo root so you need to have a built backend and frontend + for this to run locally. The server starts on port 3001 so it does not conflict with your normal dev server. +- `yarn e2e:debug` Same as above but runs the tests in chrome and does not shutdown after completion. +- `yarn e2e:dev` Same as above but does not run any tests on startup. It lets you pick a test first. + +The above commands use some utils scripts under `/e2e` that can also be used for more control. + +- `./e2e/start-server` This creates a fresh new grafana server working dir, setup's config and starts the server. It + will also kill any previously started server that is still running using pid file at `/e2e/tmp/pid`. +- `./e2e/run-suite ` Starts cypress in different modes. + +## Test Suites + +All the integration tests are located at `e2e/suite/specs`. The page objects and reusable flows are in the +`packages/grafana-e2e` package. + ## Basic concepts + Here is a good introduction to e2e best practices: https://martinfowler.com/bliki/PageObject.html. + - `Selector`: A unique identifier that is used from the e2e framework to retrieve an element from the Browser - `Page`: An abstraction for an object that contains one or more `Selectors` - `Flow`: An abstraction that contains a sequence of actions on one or more `Pages` that can be reused and shared between tests ## Basic example + Let's start with a simple example with a single selector. For simplicity, all examples are in JSX. In our example app, we have an input that we want to type some text into during our e2e test. + ```jsx harmony
- +
``` @@ -21,25 +45,28 @@ We could define a selector using `JQuery` [type selectors](https://api.jquery.co At Grafana, we use `aria-label` as our preferred way of defining selectors instead of `data-*` attributes. This also aids in accessibility. Let's add a descriptive `aria-label` to our simple example. + ```jsx harmony
- +
``` Now that we added the `aria-label` we suddenly get more information about this particular field. It's an input field that represents a username, but there it's still not really signaling that it's part of an e2e test. The next step is to create a `Page` representation in our e2e test framework to glue the test with the real implementation using the `pageFactory` function. For that function we can supply a `url` and `selectors` like in the example below: + ```typescript export const Login = pageFactory({ - url: '/login', // used when called from Login.visit() + url: "/login", // used when called from Login.visit() selectors: { - username: 'Username input field', // used when called from Login.username().type('Hello World') + username: "Username input field", // used when called from Login.username().type('Hello World') }, }); ``` The next step is to add the `Login` page to the exported const `Pages` in `packages/grafana-e2e/src/pages/index.ts` so that it appears when we type `e2e.pages` in our IDE. + ```ecmascript 6 export const Pages = { Login, @@ -49,17 +76,21 @@ export const Pages = { }; ``` + Now that we have a `Page` called `Login` in our `Pages` const we can use that to add a selector in our html like shown below and now this really signals to future developers that it is part of an e2e test. + ```jsx harmony
- +
``` The last step in our example is to use our `Login` page as part of a test. The `pageFactory` function we used before gives us two things: + - The `url` property is used whenever we call the `visit` function and is equivalent to the Cypress function [cy.visit()](https://docs.cypress.io/api/commands/visit.html#Syntax). -> Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions). + > Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions). - Any defined selector in the `selectors` property can be accessed from the `Login` page by invoking it. This is equivalent to the result of the Cypress function [cy.get(...)](https://docs.cypress.io/api/commands/get.html#Syntax). + ```ecmascript 6 describe('Login test', () => { it('Should pass', () => { @@ -73,6 +104,7 @@ describe('Login test', () => { ``` ## Advanced example + Let's take a look at an example that uses the same `selector` for multiple items in a list for instance. In this example app we have a list of data sources that we want to click on during an e2e test. ```jsx harmony @@ -80,15 +112,14 @@ Let's take a look at an example that uses the same `selector` for multiple items {dataSources.map(dataSource => (
  • -
    - {dataSource.name} -
    +
    {dataSource.name}
    • ))} ``` -``` + +```` Just as before in the basic example we'll start by creating a page abstraction using the `pageFactory` function: ```typescript @@ -98,12 +129,14 @@ export const DataSources = pageFactory({ dataSources: (dataSourceName: string) => `Data source list item ${dataSourceName}`, }, }); -``` +```` + You might have noticed that instead of a simple `string` as the `selector`, we're using a `function` that takes a string parameter as an argument and returns a formatted string using the argument. Just as before we need to add the `DataSources` page to the exported const `Pages` in `packages/grafana-e2e/src/pages/index.ts`. The next step is to use the `dataSources` selector function as in our example below: + ```jsx harmony
      {dataSources.map(dataSource => ( @@ -119,6 +152,7 @@ The next step is to use the `dataSources` selector function as in our example be ``` When this list is rendered with the data sources with names `A`, `B`, `C` the resulting html would become: + ```jsx harmony
      A @@ -134,7 +168,9 @@ When this list is rendered with the data sources with names `A`, `B`, `C` the re ``` Now we can write our test. The one thing that differs from the `Basic example` is that we pass in which data source we want to click on as an argument to the selector function: + > Best practice after calling `visit` is to always call `should` on a selector to prevent flaky tests when you try to access an element that isn't ready. For more information, refer to [Commands vs. assertions](https://docs.cypress.io/guides/core-concepts/retry-ability.html#Commands-vs-assertions). + ```ecmascript 6 describe('List test', () => { it('Clicking on data source named B', () => { @@ -147,17 +183,18 @@ describe('List test', () => { }); ``` - ## Debugging PhantomJS image rendering ### Common Error + The most common error with PhantomJs image rendering is when a PR introduces an import that has functionality that's not supported by PhantomJs. To quickly identify which new import causes this you can use a tool like `es-check`. - + 1. Run > `npx es-check es5 './public/build/*.js'` 2. Check the output for files that break es5 compatibility. -3. Lazy load the failing imports if possible. +3. Lazy load the failing imports if possible. ### Debugging + There is no easy or comprehensive way to debug PhantomJS smoke test (image rendering) failures. However, PhantomJS exposes remote debugging interface which can give you a sense of what is going wrong in the smoke test. Before performing the steps described below make sure your local Grafana instance is running: 1. Go to `tools/phantomjs` directory diff --git a/e2e/run-suite b/e2e/run-suite index b2f8dfb4229..18178f617e2 100755 --- a/e2e/run-suite +++ b/e2e/run-suite @@ -8,8 +8,23 @@ timeout 60 bash -c 'until nc -z $0 $1; do sleep 1; done' localhost $PORT echo -e "Starting Cypress scenarios" +CMD="start" +PARAMS="" +SLOWMO=0 + +if [ "$1" == "debug" ]; then + echo -e "Debug mode" + SLOWMO=1 + PARAMS="--headed --no-exit" +fi + +if [ "$1" == "dev" ]; then + echo "Dev mode" + CMD="open" +fi + cd packages/grafana-e2e -yarn start --env BASE_URL=http://localhost:$PORT,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \ +yarn $CMD --env BASE_URL=http://localhost:$PORT,CIRCLE_SHA1=$CIRCLE_SHA1,SLOWMO=$SLOWMO \ --config integrationFolder=../../e2e/suite1/specs,screenshotsFolder=../../e2e/suite1/screenshots,videosFolder=../../e2e/suite1/videos,fileServerFolder=./cypress,viewportWidth=1920,viewportHeight=1080,trashAssetsBeforeRuns=false \ - "$@" + $PARAMS diff --git a/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient (1).png b/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient (1).png new file mode 100644 index 00000000000..08b192e69a3 Binary files /dev/null and b/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient (1).png differ diff --git a/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient.png b/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient.png new file mode 100644 index 00000000000..e15f82cb14b Binary files /dev/null and b/e2e/suite1/screenshots/bar-gauge.spec.ts/bar-gauge-gradient.png differ diff --git a/e2e/suite1/screenshots/expected/bar-gauge-gradient.png b/e2e/suite1/screenshots/expected/bar-gauge-gradient.png new file mode 100644 index 00000000000..e15f82cb14b Binary files /dev/null and b/e2e/suite1/screenshots/expected/bar-gauge-gradient.png differ diff --git a/e2e/suite1/specs/bar-gauge.spec.ts b/e2e/suite1/specs/bar-gauge.spec.ts new file mode 100644 index 00000000000..204df33c69e --- /dev/null +++ b/e2e/suite1/specs/bar-gauge.spec.ts @@ -0,0 +1,18 @@ +import { e2e } from '@grafana/e2e'; + +e2e.scenario({ + describeName: 'Bar Gauge Panel', + itName: 'Bar Guage rendering e2e tests', + addScenarioDataSource: false, + addScenarioDashBoard: false, + skipScenario: false, + scenario: () => { + // open Panel Tests - Bar Gauge + e2e.flows.openDashboard('O6f11TZWk'); + + e2e() + .get('#panel-6 .bar-gauge__value') + .should('have.css', 'color', 'rgb(242, 73, 92)') + .contains('100'); + }, +}); diff --git a/jest.config.e2e.js b/jest.config.e2e.js deleted file mode 100644 index 8b3eb4ee732..00000000000 --- a/jest.config.e2e.js +++ /dev/null @@ -1,15 +0,0 @@ -require('module-alias/register'); - -module.exports = { - verbose: false, - transform: { - '^.+\\.(ts|tsx)$': 'ts-jest', - }, - moduleDirectories: ['node_modules', 'public'], - roots: ['/public/e2e-test'], - testRegex: '(\\.|/)(test)\\.(jsx?|tsx?)$', - moduleFileExtensions: ['ts', 'tsx', 'js', 'jsx', 'json'], - setupFiles: [], - globals: { 'ts-jest': { isolatedModules: true } }, - setupFilesAfterEnv: ['expect-puppeteer', '/packages/grafana-toolkit/src/e2e/install.ts'], -}; \ No newline at end of file diff --git a/package.json b/package.json index b2e2d15c54f..81d6b893722 100644 --- a/package.json +++ b/package.json @@ -155,7 +155,8 @@ "build": "grunt build", "dev": "webpack --progress --colors --config scripts/webpack/webpack.dev.js", "e2e": "./e2e/start-and-run-suite", - "e2e:debug": "SLOWMO=1 && ./e2e/start-and-run-suite --headed --no-exit", + "e2e:debug": "./e2e/start-and-run-suite debug", + "e2e:dev": "./e2e/start-and-run-suite dev", "jest": "jest --notify --watch", "jest-ci": "mkdir -p reports/junit && export JEST_JUNIT_OUTPUT_DIR=reports/junit && jest --ci --reporters=default --reporters=jest-junit --maxWorkers 2", "lint": "eslint public/app e2e/suite1 public/test --ext=.js,.ts,.tsx", diff --git a/packages/grafana-e2e/cypress/fixtures/example.json b/packages/grafana-e2e/cypress/fixtures/example.json new file mode 100644 index 00000000000..02e4254378e --- /dev/null +++ b/packages/grafana-e2e/cypress/fixtures/example.json @@ -0,0 +1,5 @@ +{ + "name": "Using fixtures to represent data", + "email": "hello@cypress.io", + "body": "Fixtures are a great way to mock data for responses to routes" +} diff --git a/yarn.lock b/yarn.lock index aed289df2ca..1108d141fa6 100644 --- a/yarn.lock +++ b/yarn.lock @@ -18135,7 +18135,13 @@ openurl@1.1.1: resolved "https://registry.yarnpkg.com/openurl/-/openurl-1.1.1.tgz#3875b4b0ef7a52c156f0db41d4609dbb0f94b387" integrity sha1-OHW0sO96UsFW8NtB1GCduw+Us4c= +<<<<<<< HEAD +opn@^5.1.0: +||||||| merged common ancestors +opn@^5.1.0, opn@^5.3.0: +======= opn@^5.5.0: +>>>>>>> origin/master version "5.5.0" resolved "https://registry.yarnpkg.com/opn/-/opn-5.5.0.tgz#fc7164fab56d235904c51c3b27da6758ca3b9bfc" integrity sha512-PqHpggC9bLV0VeWcdKhkpxY+3JTzetLSqTCWL/z/tFIbI6G8JCjondXklT1JinczLz2Xib62sSp0T/gKT4KksA==