Running Tests Interactively¶

Introduction¶

In addition to the default mode, which is to run all selected tests in a single batch and output results at the end, Testplan supports a second mode of use known as the “interactive mode”. When running in interactive mode, individual testcases and testsuites may be run on-demand and test environments may similarly be controlled on-demand.

The basic usage is to run your testplan as normal but with a “-i” flag:

$ python test_plan.py -i
...
Interactive Testplan web UI is running. Access it at:
    Local: http://localhost:39514/interactive/
    On Your Network: http://10.2.3.4:39514/interactive/

Why is this useful? The idea is that, while running all tests in a big batch is fine for fast unit tests or for running longer-running tests on a CI server, the interactive mode provides a convenient interface for engineers to start up a testing environment locally and iterate on running testcases against that environment. To assist with fast iteration, Testplan’s interactive mode has an automatic code-reloading feature which allows testcases to be tweaked, added or rewritten and then run without having to restart the main Testplan process or interrupt processes started as part of the test environment.

How code reloading works¶

Code-reloading is triggered by reload API request, there is a reload button on UI. Testplan will find modules to reload - those who have either been modified or have had their dependencies modified. For the sake of efficiency, Testplan only looks under the directory of the main script , as well as those directories where task targets are located. User can instruct testplan to look further by using extra_deps parameter of @test_plan decorator.

The steps of the reload process are described below:

When starting up the interactive mode, testplan builds a graph of dependencies used by the __main__ module. Only dependencies that are within the same directory as the testplan script are included. Dependencies are resolved using modulefinder.ModuleFinder
Dependencies are checked and reloaded on reload API. When reloading modules testplan will:
- Walk the graph of dependencies starting from __main__.
- For each module, if any of its dependencies were reloaded or the file has been modified since last reloaded, then reload this module and update __class__ of all suites it defines.
The interactive report is updated with any changed test cases.

There are a few known limitations that reload doesn’t always work under certain circumstances.

__main__ module (i.e. test_plan.py) cannot be reloaded. Because of this, it is recommended that test suites are defined in seperate modules. It is a good idea to keep the main script simple.
Testplan is not able to reload module that is under Namespace Package, the workaround is to add dummy __init__.py file.
Unlike testcases that can be added/removed/modified, adding/removing/renaming testsuite class will not take effect after reloading.
Modules or objects imported using from some_module import * syntax are not guaranteed to be properly reloaded. Such syntax is not encouraged in Python anyway as it may bring naming conflicts.
In interactive mode, all modules containing test targets will be loaded at the same time, if the modules from which tasks are scheduled have the same name, they won’t be reloaded properly.

Web UI¶

The main intended way to control Testplan in the interactive mode is via a web UI provided as part of the Testplan package. When a Testplan is run interactively, a local web server will be started to serve the UI page and links to access will be printed. The UI server listens on all IP addresses owned by your host, so may be accessed via localhost or over your LAN - useful if you run your Testplan on a Linux server but want to access the web UI from your Windows desktop for example.

Once loaded, the UI should look something like this:

If you have used the web UI for the batch mode you should find the interactive UI familiar, with some differences. The navigation structure is the same - you can click on entries on the navigation bar on the left to dig down into particular testsuites. The first major difference from the batch UI is that there are initially no testcase results - no tests have been run yet! You may run testcases by clicking the “Play” icons next to them in the navigation bar. Test results will appear in the main central section when they are available. In addition to running single testcases, you can also run all testcases in a testsuite or all testsuites in a Test environment in a similar way.

Test environments will be automatically started as required. However, you may also start or stop a test environment manually by clicking the toggle icon found only on top-level Test environments:

Configuration¶

By default, the interactive web server will listen on an ephemeral port, however you may override that by passing a port number after the “-i” option on the command-line:

$ python test_plan.py -i 4000
...
Interactive Testplan web UI is running. Access it at:
    Local: http://localhost:4000/interactive/
    On Your Network: http://10.2.3.4:4000/interactive/

Alternatively, a testplan may be programmed to always run in interactive mode. You may do this by setting the interactive_port parameter to any valid port number, including port 0 to use an ephemeral port.

@test_plan(
    name="My awesome testplan",
    interactive_port=0,
)
def main(plan):
    ...

Interactive API¶

Testplan’s web UI communicates with the backend via a RESTful HTTP API. For most people this is an implementation detail you don’t need to care about. However, for an advanced user, you may want to design your own client application which consumes the Testplan API. This could allow for an alternate UI (Testplan mobile app anyone? We accept PRs). Alternatively, the API could be used for automated control of Testplan environments and tests from another test framework entirely - if for example you are a Java developer, you could write your testcases in Java using JUnit but make API calls to a Testplan process to control your test environment before and after running tests. There are many similar possibilities!

When Testplan is started in interactive mode with debug logging enabled, as well as displaying a link to the web UI it will also display a link to view and interact with the API schema, using a generated Swagger UI. Take a look at the schema if you would like to learn more.

$ python test_plan.py -di
...
Interactive Testplan API is running. View the API schema:
    Local: http://localhost:36718/api/v1/interactive/
    On Your Network: http://10.174.117.110:36718/api/v1/interactive/
...