Release Notes

Unreleased (2026-06-03)

  • New Remote features, i.e. RemotePool and RemoteDriver, are now capable of setting up Python virtual environment on remote via newly introduced PipBasedBuilder. Please check out this updated example for its usage. Revamp certain internal implementation to align with the new remote setup design. Below are details of possible breaking changes which should only affect a small group of users:

    • testplan_path parameter in RemoteResource has been removed. Users are advised to use existing_testplan_parent within SourceTransferBuilder for specifying the Testplan import path on remote hosts.

    • _execute_cmd_remote method in RemoteResource has been removed, many other methods also have interface and implementation changed.

    • Function signature of certain utilities in process has been changed.

  • New Added --archive-runpath command-line option to automatically archive the runpath directory after test execution when tests fail or timeout. The archive is created in zstd-compressed tar format (.tar.zst) and can be configured via the --archive-runpath CLI argument. This feature helps preserve test execution artifacts for debugging failed or timed-out tests.

  • Changed Make it quicker to discard remaining tests within a Pool when requested by --skip-remaining (command line argument) or skip_strategy (programmatic option) and certain test failure occurred.

  • Changed Prevent unexpected scrolling on navigation pane during interactive mode

  • New Introducing FailedTestsExporter, a new exporter for dumping failed tests to a file. This file can be used with the --patterns-file option. Enable this exporter using the --dump-failed-tests command-line argument. It helps users track and manage failed tests efficiently during test runs.

  • Changed Upgrade plotly.js version to 3.3.1.

  • Changed Prevent parallel testplan executions on same runpath.

  • New Print search-friendly anchor in testplan log for test/suite/case start/stop.

  • Changed Fix empty rows appearing and ignored keys rendered as failing in DictMatch assertions on the UI.

  • Changed Use Paramiko SSHClient for executing command remotely.

  • Changed Fix early stop testplan exit code issue.

  • New Early stop testplan if no test is scheduled

  • New Add a sparse mode to generate parameterized test cases. Pass sparse=True in @testcase to enable it.

  • Changed Changed the behavior of subprocess I/O to use unbuffered mode

  • Changed Update orjson dumping option to allow certain non-string dictionary keys (see `this section<https://github.com/ijl/orjson?tab=readme-ov-file#opt_non_str_keys>`_ for details).

  • Changed Enhance logging for MultiTest to provide extra information when testcases execution order changed during runtime.

  • Changed Update clean_runpath_if_passed() and DriverLogCollector for the use case of invoking them inside an existing Test resource hook.

  • New Added --otel-tracing to allow tracing of test execution. When enabled, this captures information about test setup, execution, and teardown phases, allowing you to analyze bottlenecks and trace test progress. See here for more configuration details and examples.

  • Changed Several changes to report filtering command line arguments. This is a breaking change.

    • Remove command line argument --report-filter, part of its functionality can be achieved using --report-exclude instead. Please check its help message for more details.

    • Remove command line argument --omit-skipped.

    • Change the behavior of command line argument --omit-passed, now it preserves passed case entries in the report while omitting their assertions.

  • Changed Fix an issue that certain error-reporting entries being unexpectedly removed after applying report filters.

  • Changed Report filters (command line argument --omit-passed and --report-exclude) now preserve generated report entries like driver start timing information.

  • Changed Fix the issue where interactive mode was not auto-refreshing.

  • Changed Fix an issue that templated BDD step arguments were not properly rendered.

  • New Added --otel-logs flag to enable OpenTelemetry logging integration with Grafana Loki. When used alongside --otel-traces, all testplan logs are automatically captured and correlated with the active span. See here for more configuration details and examples.

  • New Added --testcase-timeout CLI option to set a default timeout for testcases without explicit timeout decorators.

  • Changed Fix an issue where the FailedTestsExporter exports the Task name instead of the test name when tasks encounter errors.

  • Changed Fix an issue where the test runs in interactive mode will be stuck at waiting when filter text is present in the search box.

  • Changed Added an edge-case check so auto-part for multitest does not split into more parts than testcases.

  • New Add a UI toggle under Preferences button to display multitest parts in merged view

  • Deprecated merge_scheduled_parts is deprecated and has no effect. Use the merge parts toggle in the UI to merge part reports. This parameter will be removed in a future release.

  • New Added an option to display dict.match and fix.match assertion results in their original order in the web ui.

  • Changed Errored-out tasks now have their test names as the names of report entries instead of the Task[...] representation.

  • Changed Fixed a bug in autopart logic producing 0 parts when the test filter includes a part specification (e.g. --patterns "MyTest - part(1/6)").

  • Changed The --patterns filter will run a Multitest as long as the test name matches even if the Multitest is empty.

  • Changed Improved driver startup timeout error messages and exception handling within error log fetching, together with other logging improvements.

  • New Resource monitor now supports collecting system load information.

  • Changed Sync remote disk after copying workspace on Remote host.

  • Changed Xfail condition parameter now accepts a failed entry with description of value None to match assertions whose description are not set.

  • New Xfail now supports a condition parameter for conditional xfail based on error message or failed assertion entry.

  • Changed Added support for HTTP OTLP exporter in the tracing module. Exporter type is configured to be HTTP if the OTEL_EXPORTER_OTLP_TRACES_ENDPOINT environment variable starts with “http://” or “https://”, else it will default to use the gRPC exporter.

  • New Add MultiTest and TestSuite timing to JUnit XML report

  • New Add MultiTest Environment start/stop timing

  • Changed Replaced the toolbar pass/fail radio filter with a multi-select checkbox filter supporting all test statuses (passed, failed, error, skipped, xfail, xpass, xpass-strict, incomplete, unknown), including correct filtering through nested entry hierarchies.

  • Changed Added an unstable counter (xpass + xfail) alongside the passed/failed counter in web UI.

  • Changed Upgrade web UI package manager configuration to use pnpm 11.

  • Changed Speed up testplan startup by skipping unnecessary source code location resolution, and fix source code location resolution for decorated testcases.

  • Changed Adjust test environment startup scheduling to avoid subprocess hangs in rare scenarios. Driver starting and stopping hooks now run outside the environment thread pool, while started/stopped readiness checks remain asynchronous.

  • Changed Start and stop dependency-managed drivers within a bounded thread pool to prevent slow driver operations from blocking others.

    This may improve overall environment startup time, but stricter dependency definitions or driver implementations may be required where shared state is not safe for concurrent execution.

  • Changed Increased precedence of XFAIL in Status. Passing test reports containing XFAIL testcases now aggregate to XFAIL instead of PASSED.

  • Changed Fix a bug in the pytest wrapper when reporting failures, and enable it to report skipped, xfail and xpass statuses.

  • Changed Setting the TESTPLAN_STRICT_PARAM_NAMES=1 environment variable now makes an over-long parametrized testcase name raise an error instead of falling back to an index-suffixed name with a warning.

  • Changed Fix AttributeError: 'function' object has no attribute '__func__' raised in get_test_context when a testcase listed in --xfail-tests (or xfail_tests=) is also skipped by a @skip_if predicate.

25.8.0 (2025-08-15)

  • Changed Fix an issue where PDFs failed to render in certain cases. Fix console output of certain assertion details.

  • Changed Upgrade gherkin-official to 32 for supporting BDD on Python 3.11 or later.

    Note that spaces are no longer allowed in gherkin tags, this could be a breaking change for BDD users. As a result, lines like @tag a @tag b in feature files should be replaced with line like @tag_a @tag_b, and line @KNOWN_TO_FAIL: some reason could be replaced with @KNOWN_TO_FAIL:some_reason.

  • New Added a timeout entry to the report to make testplan-level timeout events more visible to users. Fixed an issue where multitests incorrectly appeared as passed in stdout upon timeout. Updated the status of unfinished test suites and multitests to “incomplete”.

  • Changed Speed up interactive mode report initial load.

  • New Create new frontend components DictMatchAll and FixMatchAll.

  • Changed The wrapped driver class within RemoteDriver is now constructed only on remote host.

  • Changed Method signature check of testcases/test hooks has been relaxed to allow underscore-prefixed variants of parameters, so that linter wouldn’t raise unused variable warnings on them.

  • Changed Move certain dependencies to extras (i.e. optional dependencies) to shrink the installation size in pipelines. User need to install Testplan with extras in order to use certain features.

    • Extra interactive for interactive mode & report display through local server features, containing flask & several related packages.

    • Extra plotly for plotting with plotly feature, containing pandas & plotly.

    • Extra all for all the features above.

    For example, to use interactive mode feature, one can install Testplan with the following command:

    pip install testplan[interactive]

  • Removed Remove support for Python 3.7 and 3.8.

  • Changed Parameters auto_part_runtime_limit and plan_runtime_target of @test_plan now takes “auto”. See documentation for details.

  • Changed Make teardown_time field mandatory in the json file provided via --runtime-data cmdline arg for auto part and smart scheduling.

  • Changed Fix an issue where nested lists in dict.match and dict.match_all assertions were not fully shown in the result output.

  • Changed Temporarily restrict package dependency marshmallow to versions <4.

  • Changed Fixed fake report for UI development

  • Changed Runtime data JSON file specified through --runtime-data now accepts an optional testcase_count field, which should hold the total number of testcases from the previous run. If testcase_count is specified, estimated MultiTest execution time will be adjusted by the ratio of current run’s testcase count to that of previous run, which should bring more accurate auto-partitioning and improved testing resource utilization.

  • Changed Set timezone entries in Testplan reports to UTC when local timezone detection fails.

  • Changed Fix an issue that generated incorrect figures of driver starting/stopping timeline inside the driver-info feature.

25.3.0 (2025-03-25)

  • Changed Change Testplan exported JSON report structure to reduce report size.

    • Remove unused report entry fields.

      • fix_spec_path.

      • status_override and status_reason in case they are empty.

      • line_no, code_context and file_path if --code is not enabled.

      • env_status, part, strict_order and host depending on report category.

    • Remove unused assertion entry fields category and flag if they are DEFAULT.

    • Merge assertion entry fields utc_time and machine_time into a unix timestamp field timestamp, and store timezone info in parent Test-level report under key timezone.

    • Replace ISO 8601 time string with unix timestamp in all timer fields, and add a timezone field to Testplan-level report as well.

    • Update data structure of several serialized assertion entries.

      • Delta encode level info of flattened_dict fields of DictLog and FixLog entries.

      • Delta encode level info of comparison fields of DictMatch and FixMatch entries.

      • Delta encode level info of nested comparison fields of DictMatchAll and FixMatchAll entries, remove extra nesting of matches as well.

      • Preserve abbreviations of match status of DictMatch, FixMatch, DictMatchAll and FixMatchAll entries, i.e. p instead of Passed, f instead of Failed, i instead of Ignored.

      • Remove indices field of TableLog entries.

  • Changed Fix releaseherald documentation

  • Deprecated Support for Python 3.7 and 3.8 is deprecated and will be removed soon.

  • Changed Handle potential race condition during resource monitor termination

  • Changed Improve error logging for RemoteService; fix incorrect imitated workspace on remote due to leftover symlink from previous run.

  • Changed Remove Sphinx and other packages for building document from Testplan’s dependencies.

  • Changed Copy permission bits for the copied binary in App.

25.1.0 (2025-01-20)

  • Changed Checks if a process exists by reading the /proc/<pid>/stat.

  • Changed Support RemoteDriver in dependency graph of test environment (the dependencies parameter).

  • Changed Use lazy import for Matplotlib and move cache to runpath.

  • New Added --code flag to collect code context for the assertions. Code context one-liner will be displayed on the web UI if enabled. Note that file path information is no longer collected by default. To collect file path information, enable code context.

  • New Add a new summary page on resource view to show the task allocation per host.

  • Changed Refactor the stop logic of App driver for faster environment shutdown. Rename parameter sigint_timeout to stop_timeout. Add a new parameter stop_signal for custom stop signals, its default value None invokes terminate method to stop subprocess, i.e. sending SIGTERM signal to subprocess on Linux. Environment will fail to stop if subprocess doesn’t terminate within the default 5-second stop_timeout for graceful shutdown. Increase stop_timeout or change stop_signal (to maybe SIGKILL on Linux) could resolve this issue.

  • Changed Make sure when stop() is called on App type driver, we clean up all orphaned processes.

  • Changed If App driver times out during shutdown or leaves orphaned processes after shutdown, Testplan will now emit a warning and perform a forced cleanup instead of failing the tests.

  • Changed Increase the number of Remote worker setup thread.

  • Changed Fix interactive mode crashing issue when loading a namespace package. (It is still not supported to reload namespace packages.) SyntaxError will no longer be suppressed during interactive mode code reloading.

  • Changed Fix incorrect early stop detection logic.

  • Changed Swapped Run and Reload buttons on the interactive UI by user request.

  • Changed Use a new JSON library orjson to improve performance when using Python 3.8 or later versions.

  • Changed Limit the length of parameterization testcase name to 255 characters. If the name length exceeds 255 characters, index-suffixed names (e.g., {func_name} 1, {func_name} 2) will be used.

  • New Testplan now includes its own version in generated report.

  • Changed JSONExporter will log a “file not found” warning in the log instead of raising an exception.

  • Changed Fixed an issue where enabling Status icons crashed the report when a test was marked as XFAIL.

  • Changed Update orjson dumping option to allow serializing numpy objects.

24.9.2 (2024-09-13)

  • New First official pypi release.