From b55705bd65ce99ae598a49d4fbdc9c9c005ae144 Mon Sep 17 00:00:00 2001 From: Joao Moreno Date: Mon, 25 Sep 2017 17:36:58 +0200 Subject: [PATCH] update smoketest docs --- test/smoke/README.md | 61 +++++++++++--------------------------------- 1 file changed, 15 insertions(+), 46 deletions(-) diff --git a/test/smoke/README.md b/test/smoke/README.md index 870a904c570..74a9b7e1c21 100644 --- a/test/smoke/README.md +++ b/test/smoke/README.md @@ -1,61 +1,30 @@ -# VS Code Automated Smoke Testing +# VS Code Smoke Test -## Framework -* Smoke tests are written using the [Spectron](https://electron.atom.io/spectron/) and [Mocha](https://mochajs.org/) frameworks. -* Spectron is used to control the lifecycle of VS Code and also to query the DOM elements of VS Code renderer window using the [WebriverIO](http://webdriver.io/) API that is wrapped in Spectron API. -* Mocha is used to launch the smoke tests. +## How to run -## Code Organization -* All smoke test code is present under `/test/smoke/` folder. Code is organized into indvidual areas. Each area contains a facade to to access it functionality and a test file to test it. For e.g. `debug` area has `debug.ts` facade class that provides utility methods to access debug functionalities. It also has `debug.test.ts` that tests the debug functionalities. - -* `application.ts` provides APIs to start and stop the VS Code application. It also provides access to various utility APIs like - * client - * webclient - * screenCapturer - * workbench - -* `client.ts` class wraps WebDriverIO APIs and provides enhanced APIs for accessing DOM elements. For e.g. it has methods like `waitForElement(selector, accept)` that will query the DOM with the given selector and waits until the element is found or time out after configured time (`5s`). - -* `screenCapturer.ts` allows you capture the screenshots. Capturing is done only if argument `--screenshots` is passed while launching smoke tests. When run out of sources, screenshots are captured under `screenshots` folder in the parent directory of vscode workspace. - -* `workbench.ts` provides utlities to access workbench functionality and also access to other functionality areas like `scm`, `debug`, `editor`. **Note**: All areas should be able to be accessible from workbench either directly or through area traversal. - -### Adding new area and tests -To contribute a new smoke test area, add `${area}` folder under `./areas/`. All tests and facades related to this area should go under this folder. Newly added tests should be listed in `main.ts` as imports so that mocha can pick and run them. - -## Running "Out of Sources" ``` +# Dev npm run smoketest -``` -## Running Insiders -``` -npm run smoketest -- --build "path/to/code-insiders" -``` - -## Running Stable -``` +# Specific build npm run smoketest -- --build "path/to/code" -``` -To run 'Data Migration' tests, specify the path of the version to be migrated using `--stable` argument - -``` +# Data Migration tests npm run smoketest -- --build "path/to/code-insiders" --stable "path/to/code" ``` -By default screenshots are not captured. To run tests with screenshots use the argument `--screenshots` +The script calls mocha, so all mocha arguments should work fine. For example, use `-f Git` to only run the `Git` tests. -``` -npm run smoketest -- --screenshots -``` +By default, screenshots are not captured. To run tests with screenshots use the argument `--screenshots`. -To run a specific test suite use `-f ${suiteName}` argument +## Pitfalls -``` -npm run smoketest -- -f Git -``` +- Beware of **state**. Invariably, your test suite will leave the workspace in a different state than the one it started + with. Try to avoid these situations and strive to leave the workspace in the same state as before the test suite ran. -# Debugging +- Beware of **singletons**. This evil can manifest itself under the form of FS paths, TCP ports, IPC handles. + Whenever writing a test, make sure it can run simultaneously with any other tests and even itself. + All test suites should be able to run many times in parallel. -Update \ No newline at end of file +- Beware of **focus**. Never depend on DOM elements having focus using `.focused` classes or `:focus` pseudo-classes, since they will lose that state as soon as another window appears on top of the running VS Code window. + A safe approach which avoids this problem is to use the `waitForActiveElement` API. Many tests use this whenever they need to wait for a specific element to _have focus_. \ No newline at end of file