testplan package

Subpackages

testplan.base module

Testplan base module.

class testplan.base.Testplan(name: str, description: str | None = None, parse_cmdline: bool = True, parser: Type[TestplanParser] = <class 'testplan.parser.TestplanParser'>, interactive_port: int | None = None, abort_signals: List[Signals] | None = None, logger_level: int = 25, file_log_level: int = 10, runpath: str | Callable = <function default_runpath>, path_cleanup: bool = True, all_tasks_local: bool = False, shuffle: List[str] | None = None, shuffle_seed: float | None = None, exporters: List | None = None, stdout_style: Style = Style(passing='testcase', failing='assertion-detail'), report_dir: str = '/home/docs/checkouts/readthedocs.org/user_builds/testplan/checkouts/main/doc/en', xml_dir: str | None = None, json_path: str | None = None, http_url: str | None = None, pdf_path: str | None = None, pdf_style: Style = Style(passing='test', failing='test'), dump_failed_tests: str | None = None, failed_tests_level: FailedTestLevel = FailedTestLevel.MULTITEST, report_tags: List | None = None, report_tags_all: List | None = None, resource_monitor: bool = False, merge_scheduled_parts: bool = False, browse: bool = False, ui_port: int | None = None, web_server_startup_timeout: int = 10, test_filter: BaseFilter = <testplan.testing.filtering.Filter object>, test_sorter: BaseSorter = <testplan.testing.ordering.NoopSorter object>, test_lister: MetadataBasedLister | None = None, test_lister_output: PathLike | None = None, verbose: bool = False, debug: bool = False, timeout: int = 14400, interactive_handler: Type[TestRunnerIHandler] | None = None, extra_deps: List[str | ModuleType] | None = None, label: str | None = None, driver_info: bool = False, collect_code_context: bool = False, auto_part_runtime_limit: int | Literal['auto'] = 1800, plan_runtime_target: int | Literal['auto'] = 1800, skip_strategy: str | None = None, otel_traces: TraceLevel = TraceLevel.NONE, otel_traceparent: str | None = None, otel_logs: bool = False, **options: Any)[source]

Bases: RunnableManager

A collection of tests and tests executors with the ability to selectively execute a subset or a shuffled set of those tests.

It manages a TestRunner to execute the tests and also accepts all RunnableManagerConfig and TestRunnerConfig options.

Since it’s a manager of a TestRunner object, it exposes all TestRunner, attributes and methods like add_resource(), add(), and schedule().

Parameters:

name – Name of test plan.
description – Description of test plan.
parse_cmdline – Parse command line arguments.
parser – Command line parser.
interactive_port – Enable interactive execution mode on a port.
abort_signals – Signals to catch and trigger abort. By default, SIGINT and SIGTERM will trigger Testplan to abort.
logger_level – Logger level for stdout.
runpath – Input runpath.
path_cleanup – Clean previous runpath entries.
all_tasks_local – Schedule all tasks in local pool.
shuffle – Shuffle strategy.
shuffle_seed – Shuffle seed.
exporters – Exporters for reports creation.
stdout_style – Styling output options.
report_dir – Report directory.
xml_dir – XML output directory.
json_path – JSON output path <PATH>/*.json.
http_url – HTTP url to post JSON report.
pdf_path – PDF output path <PATH>/*.pdf.
pdf_style – PDF creation styling options.
report_tags – Matches tests marked with any of the given tags.
report_tags_all – Match tests marked with all of the given tags.
resource_monitor – Enable resource monitor.
merge_scheduled_parts – Deprecated. Use the merge parts toggle in the UI to merge part reports. This parameter is a no-op and will be removed in a future release.
browse – Open web browser to display the test report.
ui_port – Port of web server for displaying test report.
web_server_startup_timeout – Timeout for starting web server.
test_filter – Tests filtering class.
test_sorter – Tests sorting class.
test_lister – Tests listing class.
test_lister_output – listing results goes to this file, if None goes to stdout
verbose – Enable or disable verbose mode.
debug – Enable or disable debug mode.
timeout – Timeout value in seconds to kill Testplan and all child processes, default to 14400s(4h), set to 0 to disable.
interactive_handler – Handler for interactive mode execution.
extra_deps – Extra module dependencies for interactive reload, or paths of these modules.
label – Label the test report with the given name, useful to categorize or classify similar reports .
driver_info – Display driver setup / teardown time and driver interconnection information in UI report.
collect_code_context – Collects the file path, line number and code context of the assertions.
auto_part_runtime_limit – Maximum estimated runtime of an automatically split MultiTest part.
plan_runtime_target – Targeted runtime for the entire Testplan.
skip_strategy – Skip the execution of rest testcases under certain conditions.
otel_traces – Enable OpenTelemetry tracing at specified level.
otel_traceparent – Optional traceparent for OpenTelemetry traces.

Param:

file_log_level: Logger level for file.

CONFIG: alias of TestplanConfig

STATUS: alias of EntityStatus

abort() → None: Default abort policy. First abort all dependencies and then itself.

abort_dependencies() → Generator[Runnable, None, None]: Dependencies to be aborted first.

property aborted: bool: Returns if entity was aborted.

aborting() → None: Suppressing not implemented debug log by parent class.

property active: bool: Expose the runnable active attribute.

property args: Namespace: Parsed arguments.

property cfg: Config: Expose the runnable configuration object.

context_input(exclude: list | None = None) → Dict[str, Any]: All attr of self in a dict for context resolution

define_runpath() → None: Define runpath directory based on parent object and configuration.

enrich_options(options: Dict[str, Any]) → Dict[str, Any][source]: Enrich the options using parsed command line arguments. The command line arguments will override any explicit programmatic declaration for a given keyword.

classmethod filter_locals(local_vars: Dict[str, Any]) → Dict[str, Any]

Filter out init params of None value, they will take default value defined in its ConfigOption object; also filter out special vars that are not init params from local_vars.

Parameters:: local_vars

property logger: TestplanLogger: logger object

classmethod main_wrapper(name: str, description: str | None = None, parse_cmdline: bool = True, parser: ~typing.Type[~testplan.parser.TestplanParser] = <class 'testplan.parser.TestplanParser'>, interactive_port: int | None = None, abort_signals: ~typing.List[~signal.Signals] | None = None, logger_level: int = 25, file_log_level: int = 10, runpath: str | ~typing.Callable[[...], str] = <function default_runpath>, path_cleanup: bool = True, all_tasks_local: bool = False, shuffle: ~typing.List[str] | None = None, shuffle_seed: float | None = None, exporters: ~typing.List[~typing.Any] | None = None, stdout_style: ~testplan.report.testing.styles.Style = Style(passing='testcase', failing='assertion-detail'), report_dir: str = '/home/docs/checkouts/readthedocs.org/user_builds/testplan/checkouts/main/doc/en', xml_dir: str | None = None, json_path: str | None = None, http_url: str | None = None, pdf_path: str | None = None, pdf_style: ~testplan.report.testing.styles.Style = Style(passing='test', failing='test'), dump_failed_tests: str | None = None, failed_tests_level: ~testplan.exporters.testing.failed_tests.FailedTestLevel = FailedTestLevel.MULTITEST, report_tags: ~typing.List[~typing.Any] | None = None, report_tags_all: ~typing.List[~typing.Any] | None = None, resource_monitor: bool = False, merge_scheduled_parts: bool = False, browse: bool = False, ui_port: int | None = None, web_server_startup_timeout: int = 10, test_filter: ~testplan.testing.filtering.BaseFilter = <testplan.testing.filtering.Filter object>, test_sorter: ~testplan.testing.ordering.BaseSorter = <testplan.testing.ordering.NoopSorter object>, test_lister: ~testplan.testing.listing.MetadataBasedLister | None = None, test_lister_output: ~os.PathLike[str] | None = None, verbose: bool = False, debug: bool = False, timeout: int = 14400, interactive_handler: ~typing.Type[TestRunnerIHandler] | None = None, extra_deps: ~typing.List[str | ~types.ModuleType] | None = None, label: str | None = None, driver_info: bool = False, collect_code_context: bool = False, auto_part_runtime_limit: int | ~typing.Literal['auto'] = 1800, plan_runtime_target: int | ~typing.Literal['auto'] = 1800, skip_strategy: str | None = None, otel_traces: ~testplan.common.utils.observability.trace_level.TraceLevel = TraceLevel.NONE, otel_traceparent: str | None = None, otel_logs: bool = False, **options: ~typing.Any) → Callable[[...], Callable[[...], TestplanResult]][source]

Decorator that will be used for wrapping main methods in test scripts.

It accepts all arguments of a Testplan entity.

make_runpath_dirs() → None: Creates runpath related directories.

property parent: Entity | None: Returns parent Entity.

property parser: TestplanParser: Returns a new command line parser.

pause() → None: Pauses entity execution.

pausing() → None: Pause the runnable execution.

property processed_args: Dict[str, Any]: Processed parsed arguments.

property report: ReportLink: A handle to access the report via recursive parent

resume() → None: Resumes entity execution.

resuming() → None: Resume the runnable execution.

run() → TestplanResult[source]

Runs the tests added and returns the result object. Also handles usr1 and usr2 signals.

Returns:: Result containing tests and execution steps results.
Return type:: TestplanResult

property runnable: Runnable: Runnable instance.

property runpath: str: Expose the runnable runpath.

property scratch: str: Path to be used for temp files by entity.

property status: EntityStatus: Expose the runnable status.

property timer: Timer

uid() → str: Unique identifier of self.

wait(target_status: str | None, timeout: int | None = None) → None

Wait until objects status becomes target status.

Parameters:

target_status (str) – expected status
timeout (int or NoneType) – timeout in seconds

class testplan.base.TestplanConfig(**options: Any)[source]

Bases: RunnableManagerConfig, TestRunnerConfig

Configuration object for Testplan entity.

classmethod build_schema() → Schema: Build a validation schema using the config options defined in this class and its parent classes.

denormalize() → Config: Create new config object that inherits all explicit attributes from its parents as well.

get_local(name: str, default: Any = None) → Any: Returns a local config setting (not from container)

classmethod get_options() → Dict[Any, Any][source]: Additional and updated config options for Testplan class.

ignore_extra_keys = True

property parent: Config | None: Returns the parent configuration.

set_local(name: str, value: Any) → None: set without any check

class testplan.base.TestplanMock(*args: Any, **kwargs: Any)[source]

Bases: Testplan

A mock Testplan class for testing purpose. It is recommended to use mockplan fixture defined in conftest.py if you can. Only use this when necessary, e.g. you need to override default parameters.

CONFIG: alias of TestplanConfig

STATUS: alias of EntityStatus

abort() → None: Default abort policy. First abort all dependencies and then itself.

abort_dependencies() → Generator[Runnable, None, None]: Dependencies to be aborted first.

property aborted: bool: Returns if entity was aborted.

aborting() → None: Suppressing not implemented debug log by parent class.

property active: bool: Expose the runnable active attribute.

property args: Namespace: Parsed arguments.

property cfg: Config: Expose the runnable configuration object.

context_input(exclude: list | None = None) → Dict[str, Any]: All attr of self in a dict for context resolution

define_runpath() → None: Define runpath directory based on parent object and configuration.

enrich_options(options: Dict[str, Any]) → Dict[str, Any]: Enrich the options using parsed command line arguments. The command line arguments will override any explicit programmatic declaration for a given keyword.

classmethod filter_locals(local_vars: Dict[str, Any]) → Dict[str, Any]

Filter out init params of None value, they will take default value defined in its ConfigOption object; also filter out special vars that are not init params from local_vars.

Parameters:: local_vars

property logger: TestplanLogger: logger object

classmethod main_wrapper(name: str, description: str | None = None, parse_cmdline: bool = True, parser: ~typing.Type[~testplan.parser.TestplanParser] = <class 'testplan.parser.TestplanParser'>, interactive_port: int | None = None, abort_signals: ~typing.List[~signal.Signals] | None = None, logger_level: int = 25, file_log_level: int = 10, runpath: str | ~typing.Callable[[...], str] = <function default_runpath>, path_cleanup: bool = True, all_tasks_local: bool = False, shuffle: ~typing.List[str] | None = None, shuffle_seed: float | None = None, exporters: ~typing.List[~typing.Any] | None = None, stdout_style: ~testplan.report.testing.styles.Style = Style(passing='testcase', failing='assertion-detail'), report_dir: str = '/home/docs/checkouts/readthedocs.org/user_builds/testplan/checkouts/main/doc/en', xml_dir: str | None = None, json_path: str | None = None, http_url: str | None = None, pdf_path: str | None = None, pdf_style: ~testplan.report.testing.styles.Style = Style(passing='test', failing='test'), dump_failed_tests: str | None = None, failed_tests_level: ~testplan.exporters.testing.failed_tests.FailedTestLevel = FailedTestLevel.MULTITEST, report_tags: ~typing.List[~typing.Any] | None = None, report_tags_all: ~typing.List[~typing.Any] | None = None, resource_monitor: bool = False, merge_scheduled_parts: bool = False, browse: bool = False, ui_port: int | None = None, web_server_startup_timeout: int = 10, test_filter: ~testplan.testing.filtering.BaseFilter = <testplan.testing.filtering.Filter object>, test_sorter: ~testplan.testing.ordering.BaseSorter = <testplan.testing.ordering.NoopSorter object>, test_lister: ~testplan.testing.listing.MetadataBasedLister | None = None, test_lister_output: ~os.PathLike[str] | None = None, verbose: bool = False, debug: bool = False, timeout: int = 14400, interactive_handler: ~typing.Type[TestRunnerIHandler] | None = None, extra_deps: ~typing.List[str | ~types.ModuleType] | None = None, label: str | None = None, driver_info: bool = False, collect_code_context: bool = False, auto_part_runtime_limit: int | ~typing.Literal['auto'] = 1800, plan_runtime_target: int | ~typing.Literal['auto'] = 1800, skip_strategy: str | None = None, otel_traces: ~testplan.common.utils.observability.trace_level.TraceLevel = TraceLevel.NONE, otel_traceparent: str | None = None, otel_logs: bool = False, **options: ~typing.Any) → Callable[[...], Callable[[...], TestplanResult]]

Decorator that will be used for wrapping main methods in test scripts.

It accepts all arguments of a Testplan entity.

make_runpath_dirs() → None: Creates runpath related directories.

property parent: Entity | None: Returns parent Entity.

property parser: TestplanParser: Returns a new command line parser.

pause() → None: Pauses entity execution.

pausing() → None: Pause the runnable execution.

property processed_args: Dict[str, Any]: Processed parsed arguments.

property report: ReportLink: A handle to access the report via recursive parent

resume() → None: Resumes entity execution.

resuming() → None: Resume the runnable execution.

run() → TestplanResult

Runs the tests added and returns the result object. Also handles usr1 and usr2 signals.

Returns:: Result containing tests and execution steps results.
Return type:: TestplanResult

property runnable: Runnable: Runnable instance.

property runpath: str: Expose the runnable runpath.

property scratch: str: Path to be used for temp files by entity.

property status: EntityStatus: Expose the runnable status.

property timer: Timer

uid() → str: Unique identifier of self.

wait(target_status: str | None, timeout: int | None = None) → None

Wait until objects status becomes target status.

Parameters:

target_status (str) – expected status
timeout (int or NoneType) – timeout in seconds

class testplan.base.TestplanResult[source]

Bases: TestRunnerResult

Result object of a Testplan runnable manager entity.

property exit_code: int: System exit code based on successful run.

exporter_results: List[Any]

report: TestReport

run: bool | Exception

step_results: OrderedDict[str, Any]

property success: bool: Run was successful.

test_results: OrderedDict[str, TestResult]

testplan.base.default_runpath_mock(entity: Entity) → str[source]: To avoid runpath collision in testing

testplan.base.pdb_drop_handler(sig: int, frame: FrameType | None) → None[source]: Drop into pdb

testplan.base.test_plan(name: str, description: str | None = None, parse_cmdline: bool = True, parser: ~typing.Type[~testplan.parser.TestplanParser] = <class 'testplan.parser.TestplanParser'>, interactive_port: int | None = None, abort_signals: ~typing.List[~signal.Signals] | None = None, logger_level: int = 25, file_log_level: int = 10, runpath: str | ~typing.Callable[[...], str] = <function default_runpath>, path_cleanup: bool = True, all_tasks_local: bool = False, shuffle: ~typing.List[str] | None = None, shuffle_seed: float | None = None, exporters: ~typing.List[~typing.Any] | None = None, stdout_style: ~testplan.report.testing.styles.Style = Style(passing='testcase', failing='assertion-detail'), report_dir: str = '/home/docs/checkouts/readthedocs.org/user_builds/testplan/checkouts/main/doc/en', xml_dir: str | None = None, json_path: str | None = None, http_url: str | None = None, pdf_path: str | None = None, pdf_style: ~testplan.report.testing.styles.Style = Style(passing='test', failing='test'), dump_failed_tests: str | None = None, failed_tests_level: ~testplan.exporters.testing.failed_tests.FailedTestLevel = FailedTestLevel.MULTITEST, report_tags: ~typing.List[~typing.Any] | None = None, report_tags_all: ~typing.List[~typing.Any] | None = None, resource_monitor: bool = False, merge_scheduled_parts: bool = False, browse: bool = False, ui_port: int | None = None, web_server_startup_timeout: int = 10, test_filter: ~testplan.testing.filtering.BaseFilter = <testplan.testing.filtering.Filter object>, test_sorter: ~testplan.testing.ordering.BaseSorter = <testplan.testing.ordering.NoopSorter object>, test_lister: ~testplan.testing.listing.MetadataBasedLister | None = None, test_lister_output: ~os.PathLike[str] | None = None, verbose: bool = False, debug: bool = False, timeout: int = 14400, interactive_handler: ~typing.Type[TestRunnerIHandler] | None = None, extra_deps: ~typing.List[str | ~types.ModuleType] | None = None, label: str | None = None, driver_info: bool = False, collect_code_context: bool = False, auto_part_runtime_limit: int | ~typing.Literal['auto'] = 1800, plan_runtime_target: int | ~typing.Literal['auto'] = 1800, skip_strategy: str | None = None, otel_traces: ~testplan.common.utils.observability.trace_level.TraceLevel = TraceLevel.NONE, otel_traceparent: str | None = None, otel_logs: bool = False, **options: ~typing.Any) → Callable[[...], Callable[[...], TestplanResult]]

Decorator that will be used for wrapping main methods in test scripts.

It accepts all arguments of a Testplan entity.

testplan.defaults module

All default values that will be shared between config objects go here.

testplan.environment module

Module containing environments related classes.

class testplan.environment.EnvironmentCreator(uid, resources=None)[source]

Bases: object

Environment creator interface.

add_resource(resource)[source]: Add a resource to the future created environment.

create(parent)[source]: Create a new environment containing added resources.

uid()[source]: Environment uid.

class testplan.environment.Environments(uid='environments', **options)[source]

Bases: Resource

Environments holder resource.

Also inherits all Resource options.

abort_dependencies()[source]: Abort all resources on all environments.

aborting()[source]: Abort logic.

add(env, uid)[source]: Add an environment.

delete(uid)[source]: Delete an environment.

property envs: Returns all added environments.

starting()[source]: Start all added environments.

stopping()[source]: Stop all added environments.

class testplan.environment.LocalEnvironment(uid, resources=None)[source]

Bases: EnvironmentCreator

Creator class of a local environment.

add_resource(resource)[source]: Add a resource to the future created environment.

create(parent)[source]: Create a new environment containing added resources.

testplan.parser module

Classes that parse command-line arguments used to control testplan behaviour. This module encodes the argument and option names, types, and behaviours.

class testplan.parser.HelpParser(prog=None, usage=None, description=None, epilog=None, parents=[], formatter_class=<class 'argparse.HelpFormatter'>, prefix_chars='-', fromfile_prefix_chars=None, argument_default=None, conflict_handler='error', add_help=True, allow_abbrev=True, exit_on_error=True, *, suggest_on_error=False, color=True)[source]

Bases: ArgumentParser

Extends ArgumentParser in order to print the help message upon failure.

error(message: str) → NoReturn[source]

Overrides error method to print error and display help message.

Parameters:: message – the parsing error message

class testplan.parser.LogLevelAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None, deprecated=False)[source]

Bases: Action

Custom parser action to convert from a string log level to its int value, e.g. “DEBUG” -> 10. The level can also be specified as “NONE”, which will be stored internally as None.

LEVELS = {'CRITICAL': 50, 'DEBUG': 10, 'ERROR': 40, 'INFO': 20, 'NONE': None, 'USER_INFO': 25, 'WARNING': 30}

class testplan.parser.TestplanParser(name: str, default_options: Dict)[source]

Bases: object

Wrapper around argparse.ArgumentParser, adds extra step for processing arguments, useful when there are cross-dependencies between them.

add_arguments(parser: HelpParser) → None[source]

Virtual method to be overridden by custom parsers.

Parameters:: parser – parser instance

generate_parser() → HelpParser[source]: Generates an argparse.ArgumentParser instance.

parse_args() → Namespace[source]: Generates the parser & return parsed command line args.

process_args(namespace: Namespace) → Dict[source]

Overrides this method to add extra argument processing logic.

Can be used for interdependent argument processing.

Testplan uses the result dictionary to initialize the configuration.

Parameters:: namespace – namespace of parsed arguments
Returns:: initial configuration

Module contents

Module containing Testplan main class.