Release Notes
~~~~~~~~~~~~~

.. role:: changed
.. role:: new
.. role:: deprecated
.. role:: removed

.. releaseherald_insert

.. _rev_Unreleased:


Unreleased (2026-06-03)
-----------------------

* :new:`New` Remote features, i.e. :py:class:`~testplan.runners.pools.remote.RemotePool` and :py:class:`~testplan.common.remote.remote_driver.RemoteDriver`, are now capable of setting up Python virtual environment on remote
  via newly introduced :py:class:`~testplan.common.remote.remote_runtime.PipBasedBuilder`. Please check out :ref:`this updated example <example_pool_remote>` for its usage.
  Revamp certain internal implementation to align with the new remote setup design. Below are details of possible **breaking changes** which should only affect a small group of users:

  * ``testplan_path`` parameter in :py:class:`~testplan.common.remote.remote_resource.RemoteResource` has been removed. Users are advised to use ``existing_testplan_parent`` within :py:class:`~testplan.common.remote.remote_runtime.SourceTransferBuilder` for specifying the Testplan import path on remote hosts.
  * ``_execute_cmd_remote`` method in :py:class:`~testplan.common.utils.remote.RemoteResource` has been removed, many other methods also have interface and implementation changed.
  * Function signature of certain utilities in :py:mod:`~testplan.common.utils.process` has been changed.

* :new:`New` Added ``--archive-runpath`` command-line option to automatically archive the runpath directory after test execution when tests fail or timeout. The archive is created in zstd-compressed tar format (.tar.zst) and can be configured via the ``--archive-runpath`` CLI argument. This feature helps preserve test execution artifacts for debugging failed or timed-out tests.
* :changed:`Changed` Make it quicker to discard remaining tests within a :py:class:`~testplan.runners.pools.base.Pool` when requested by ``--skip-remaining`` (command line argument) or ``skip_strategy`` (programmatic option) and certain test failure occurred.
* :changed:`Changed` Prevent unexpected scrolling on navigation pane during interactive mode
* :new:`New` Introducing ``FailedTestsExporter``, a new exporter for dumping failed tests to a file. This file can be used with the ``--patterns-file`` option. Enable this exporter using the ``--dump-failed-tests`` command-line argument. It helps users track and manage failed tests efficiently during test runs.
* :changed:`Changed` Upgrade ``plotly.js`` version to ``3.3.1``.
* :changed:`Changed` Prevent parallel testplan executions on same runpath.
* :new:`New` Print search-friendly anchor in testplan log for test/suite/case start/stop.
* :changed:`Changed` Fix empty rows appearing and ignored keys rendered as failing in DictMatch assertions on the UI.
* :changed:`Changed` Use Paramiko SSHClient for executing command remotely.

* :changed:`Changed` Fix early stop testplan exit code issue.
* :new:`New` Early stop testplan if no test is scheduled
* :new:`New` Add a sparse mode to generate parameterized test cases. Pass ``sparse=True`` in ``@testcase`` to enable it.
* :changed:`Changed` Changed the behavior of subprocess I/O to use unbuffered mode
* :changed:`Changed` Update ``orjson`` dumping option to allow certain non-string dictionary keys (see `this section<https://github.com/ijl/orjson?tab=readme-ov-file#opt_non_str_keys>`_ for details).
* :changed:`Changed` Enhance logging for :py:class:`~testplan.testing.multitest.MultiTest` to provide extra information when testcases execution order changed during runtime.
* :changed:`Changed` Update :py:func:`~testplan.common.utils.helper.clean_runpath_if_passed` and :py:class:`~testplan.common.utils.helper.DriverLogCollector` for the use case of invoking them inside an existing :py:class:`~testplan.testing.base.Test` resource hook.
* :new:`New` Added ``--otel-tracing`` to allow tracing of test execution. When enabled, this captures information about test setup, execution, and teardown phases, allowing you to analyze bottlenecks and trace test progress. See :ref:`here <observability>` for more configuration details and examples.
* :changed:`Changed` Several changes to report filtering command line arguments. **This is a breaking change.**

  * Remove command line argument ``--report-filter``, part of its functionality can be achieved using ``--report-exclude`` instead. Please check its help message for more details.
  * Remove command line argument ``--omit-skipped``.
  * Change the behavior of command line argument ``--omit-passed``, now it preserves passed case entries in the report while omitting their assertions.
* :changed:`Changed` Fix an issue that certain error-reporting entries being unexpectedly removed after applying report filters.
* :changed:`Changed` Report filters (command line argument ``--omit-passed`` and ``--report-exclude``) now preserve generated report entries like driver start timing information.
* :changed:`Changed` Fix the issue where interactive mode was not auto-refreshing.
* :changed:`Changed` Fix an issue that templated BDD step arguments were not properly rendered.

* :new:`New` Added ``--otel-logs`` flag to enable OpenTelemetry logging integration with Grafana Loki. When used alongside ``--otel-traces``, all testplan logs are automatically captured and correlated with the active span. See :ref:`here <observability>` for more configuration details and examples.
* :new:`New` Added ``--testcase-timeout`` CLI option to set a default timeout for testcases without explicit timeout decorators.

* :changed:`Changed` Fix an issue where the ``FailedTestsExporter`` exports the Task name instead of the test name when tasks encounter errors.
* :changed:`Changed` Fix an issue where the test runs in interactive mode will be stuck at waiting when filter text is present in the search box.
* :changed:`Changed` Added an edge-case check so auto-part for multitest does not split into more parts than testcases.
* :new:`New` Add a UI toggle under Preferences button to display multitest parts in merged view
* :deprecated:`Deprecated` ``merge_scheduled_parts`` is deprecated and has no effect. Use the merge parts toggle in the UI to merge part reports. This parameter will be removed in a future release.
* :new:`New` Added an option to display ``dict.match`` and ``fix.match`` assertion results in their original order in the web ui.
* :changed:`Changed` Errored-out tasks now have their test names as the names of report entries instead of the ``Task[...]`` representation.
* :changed:`Changed` Fixed a bug in autopart logic producing 0 parts when the test filter includes a part specification (e.g. ``--patterns "MyTest - part(1/6)"``).
* :changed:`Changed` The ``--patterns`` filter will run a Multitest as long as the test name matches even if the Multitest is empty.
* :changed:`Changed` Improved driver startup timeout error messages and exception handling within error log fetching, together with other logging improvements.

* :new:`New` Resource monitor now supports collecting system load information.
* :changed:`Changed` Sync remote disk after copying workspace on Remote host.
* :changed:`Changed` :ref:`Xfail` ``condition`` parameter now accepts a ``failed`` entry with ``description`` of value ``None`` to match assertions whose description are not set.

* :new:`New` :ref:`Xfail` now supports a ``condition`` parameter for conditional xfail based on error message or failed assertion entry.
* :changed:`Changed` Added support for HTTP OTLP exporter in the tracing module. Exporter type is configured to be HTTP if the ``OTEL_EXPORTER_OTLP_TRACES_ENDPOINT`` environment variable starts with "http://" or "https://", else it will default to use the gRPC exporter.
* :new:`New` Add MultiTest and TestSuite timing to JUnit XML report

* :new:`New` Add MultiTest Environment start/stop timing

* :changed:`Changed` Replaced the toolbar pass/fail radio filter with a multi-select checkbox filter supporting all test statuses (passed, failed, error, skipped, xfail, xpass, xpass-strict, incomplete, unknown), including correct filtering through nested entry hierarchies.
* :changed:`Changed` Added an unstable counter (xpass + xfail) alongside the passed/failed counter in web UI.

* :changed:`Changed` Upgrade web UI package manager configuration to use pnpm 11.

* :changed:`Changed` Speed up testplan startup by skipping unnecessary source code location resolution, and fix source code location resolution for decorated testcases.
* :changed:`Changed` Adjust test environment startup scheduling to avoid subprocess hangs in rare scenarios. Driver ``starting`` and ``stopping`` hooks now run outside the environment thread pool, while started/stopped readiness checks remain asynchronous.

* :changed:`Changed` Start and stop dependency-managed drivers within a bounded thread pool to prevent slow driver operations from blocking others.

  This may improve overall environment startup time, but stricter dependency definitions or driver implementations may be required where shared state is not safe for concurrent execution.

* :changed:`Changed` Increased precedence of XFAIL in :py:class:`~testplan.common.report.base.Status`. Passing test reports containing XFAIL testcases now aggregate to XFAIL instead of PASSED.

* :changed:`Changed` Fix a bug in the pytest wrapper when reporting failures, and enable it to report skipped, xfail and xpass statuses.

* :changed:`Changed` Setting the ``TESTPLAN_STRICT_PARAM_NAMES=1`` environment variable now makes an over-long parametrized testcase name raise an error instead of falling back to an index-suffixed name with a warning.

* :changed:`Changed` Fix ``AttributeError: 'function' object has no attribute '__func__'`` raised in ``get_test_context`` when a testcase listed in ``--xfail-tests`` (or ``xfail_tests=``) is also skipped by a ``@skip_if`` predicate.


.. _rev_25.8.0:


25.8.0 (2025-08-15)
-------------------

* :changed:`Changed` Fix an issue where PDFs failed to render in certain cases. Fix console output of certain assertion details.
* :changed:`Changed` Upgrade ``gherkin-official`` to ``32`` for supporting BDD on Python 3.11 or later.

  Note that spaces are no longer allowed in gherkin tags, **this could be a breaking change for BDD users.** As a result, lines like ``@tag a @tag b`` in feature files should be replaced with line like ``@tag_a @tag_b``, and line ``@KNOWN_TO_FAIL: some reason`` could be replaced with ``@KNOWN_TO_FAIL:some_reason``.
* :new:`New` Added a timeout entry to the report to make testplan-level timeout events more visible to users. Fixed an issue where multitests incorrectly appeared as passed in stdout upon timeout. Updated the status of unfinished test suites and multitests to "incomplete".
* :changed:`Changed` Speed up interactive mode report initial load.

* :new:`New` Create new frontend components ``DictMatchAll`` and ``FixMatchAll``.
* :changed:`Changed` The wrapped driver class within :py:class:`RemoteDriver <testplan.common.remote.remote_driver.RemoteDriver>` is now constructed only on remote host.
* :changed:`Changed` Method signature check of testcases/test hooks has been relaxed to allow underscore-prefixed variants of parameters, so that linter wouldn't raise unused variable warnings on them.
* :changed:`Changed` Move certain dependencies to extras (i.e. optional dependencies) to shrink the installation size in pipelines. User need to install Testplan with extras in order to use certain features.

  * Extra ``interactive`` for interactive mode & report display through local server features, containing ``flask`` & several related packages.
  * Extra ``plotly`` for plotting with plotly feature, containing ``pandas`` & ``plotly``.
  * Extra ``all`` for all the features above.

  For example, to use interactive mode feature, one can install Testplan with the following command:

  .. code-block:: bash

      pip install testplan[interactive]

* :removed:`Removed` Remove support for Python 3.7 and 3.8.
* :changed:`Changed` Parameters ``auto_part_runtime_limit`` and ``plan_runtime_target`` of @test_plan now takes "auto". See :ref:`documentation <auto_part>` for details.

* :changed:`Changed` Make ``teardown_time`` field mandatory in the json file provided via ``--runtime-data`` cmdline arg for auto part and smart scheduling.

* :changed:`Changed` Fix an issue where nested lists in ``dict.match`` and ``dict.match_all`` assertions were not fully shown in the result output.
* :changed:`Changed` Temporarily restrict package dependency ``marshmallow`` to versions ``<4``.
* :changed:`Changed` Fixed fake report for UI development

* :changed:`Changed` Runtime data JSON file specified through ``--runtime-data`` now accepts an optional ``testcase_count`` field, which should hold the total number of testcases from the previous run. If ``testcase_count`` is specified, estimated MultiTest execution time will be adjusted by the ratio of current run's testcase count to that of previous run, which should bring more accurate auto-partitioning and improved testing resource utilization.
* :changed:`Changed` Set ``timezone`` entries in Testplan reports to ``UTC`` when local timezone detection fails.
* :changed:`Changed` Fix an issue that generated incorrect figures of driver starting/stopping timeline inside the driver-info feature.


.. _rev_25.3.0:


25.3.0 (2025-03-25)
-------------------

* :changed:`Changed` Change Testplan exported JSON report structure to reduce report size.

  * Remove unused report entry fields.
      * ``fix_spec_path``.
      * ``status_override`` and ``status_reason`` in case they are empty.
      * ``line_no``, ``code_context`` and ``file_path`` if ``--code`` is not enabled.
      * ``env_status``, ``part``, ``strict_order`` and ``host`` depending on report category.
  * Remove unused assertion entry fields ``category`` and ``flag`` if they are ``DEFAULT``.
  * Merge assertion entry fields ``utc_time`` and ``machine_time`` into a unix timestamp field ``timestamp``, and store timezone info in parent Test-level report under key ``timezone``.
  * Replace ISO 8601 time string with unix timestamp in all ``timer`` fields, and add a ``timezone`` field to Testplan-level report as well.
  * Update data structure of several serialized assertion entries.
      * Delta encode level info of ``flattened_dict`` fields of ``DictLog`` and ``FixLog`` entries.
      * Delta encode level info of ``comparison`` fields of ``DictMatch`` and ``FixMatch`` entries.
      * Delta encode level info of nested ``comparison`` fields of ``DictMatchAll`` and ``FixMatchAll`` entries, remove extra nesting of ``matches`` as well.
      * Preserve abbreviations of match status of ``DictMatch``, ``FixMatch``, ``DictMatchAll`` and ``FixMatchAll`` entries, i.e. ``p`` instead of ``Passed``, ``f`` instead of ``Failed``, ``i`` instead of ``Ignored``.
      * Remove ``indices`` field of ``TableLog`` entries.

* :changed:`Changed` Fix releaseherald documentation
* :deprecated:`Deprecated` Support for Python 3.7 and 3.8 is deprecated and will be removed soon.
* :changed:`Changed` Handle potential race condition during resource monitor termination
* :changed:`Changed` Improve error logging for :py:class:`~testplan.common.remote.remote_service.RemoteService`; fix incorrect imitated workspace on remote due to leftover symlink from previous run.
* :changed:`Changed` Remove Sphinx and other packages for building document from Testplan's dependencies.
* :changed:`Changed` Copy permission bits for the copied binary in App.

.. _rev_25.1.0:


25.1.0 (2025-01-20)
-------------------

* :changed:`Changed` Checks if a process exists by reading the `/proc/<pid>/stat`.
* :changed:`Changed` Support :py:class:`RemoteDriver <testplan.common.remote.remote_driver.RemoteDriver>` in dependency graph of test environment (the ``dependencies`` parameter).
* :changed:`Changed` Use lazy import for Matplotlib and move cache to runpath.
* :new:`New` Added ``--code`` flag to collect code context for the assertions. Code context one-liner will be displayed on the web UI if enabled.
  Note that file path information is no longer collected by default. To collect file path information, enable code context.
* :new:`New` Add a new summary page on resource view to show the task allocation per host.
* :changed:`Changed` Refactor the stop logic of :py:class:`App <~testplan.testing.multitest.driver.app.App>` driver for faster environment shutdown. Rename parameter ``sigint_timeout`` to ``stop_timeout``. Add a new parameter ``stop_signal`` for custom stop signals, its default value ``None`` invokes ``terminate`` method to stop subprocess, i.e. sending ``SIGTERM`` signal to subprocess on Linux.
  **Environment will fail to stop if subprocess doesn't terminate within the default 5-second** ``stop_timeout`` **for graceful shutdown**. Increase ``stop_timeout`` or change ``stop_signal`` (to maybe ``SIGKILL`` on Linux) could resolve this issue.
* :changed:`Changed` Make sure when stop() is called on App type driver, we clean up all orphaned processes.

* :changed:`Changed` If :py:class:`App <~testplan.testing.multitest.driver.app.App>` driver times out during shutdown or leaves orphaned processes after shutdown, Testplan will now emit a warning and perform a forced cleanup instead of failing the tests.
* :changed:`Changed` Increase the number of Remote worker setup thread.
* :changed:`Changed` Fix interactive mode crashing issue when loading a namespace package. (It is still not supported to reload namespace packages.) ``SyntaxError`` will no longer be suppressed during interactive mode code reloading.
* :changed:`Changed` Fix incorrect early stop detection logic.
* :changed:`Changed` Swapped Run and Reload buttons on the interactive UI by user request.
* :changed:`Changed` Use a new JSON library ``orjson`` to improve performance when using Python 3.8 or later versions.
* :changed:`Changed` Limit the length of parameterization testcase name to 255 characters. If the name length exceeds 255 characters, index-suffixed names (e.g., ``{func_name} 1``, ``{func_name} 2``) will be used.
* :new:`New` Testplan now includes its own version in generated report.
* :changed:`Changed` ``JSONExporter`` will log a "file not found" warning in the log instead of raising an exception.
* :changed:`Changed` Fixed an issue where enabling Status icons crashed the report when a test was marked as XFAIL.
* :changed:`Changed` Update ``orjson`` dumping option to allow serializing ``numpy`` objects.

.. _rev_24.9.2:


24.9.2 (2024-09-13)
-------------------

* :new:`New` First official pypi release.