testplan.report.testing package

Submodules

testplan.report.testing.base module

Report classes that will store the test results.

Assuming we have a Testplan setup like this: .. code-block:: python

Testplan MyPlan

Multitest A

Suite A-1

TestCase test_method_a_1_x TestCase test_method_a_1_y TestCase (parametrized, with 3 scenarios) test_method_a_1_z

Suite A-2

Testcase test_method_a_2_x

Multitest B

Suite B-1

Testcase test_method_b_1_x

GTest C

We will have a report tree like: .. code-block:: python

TestReport(name=’MyPlan’)

TestGroupReport(name=’A’, category=’Multitest’)

TestGroupReport(name=’A-1’, category=’TestSuite’)

TestCaseReport(name=’test_method_a_1_x’) TestCaseReport(name=’test_method_a_1_y’) TestGroupReport(name=’test_method_a_1_z’, category=’parametrization’)

TestCaseReport(name=’test_method_a_1_z_1’) TestCaseReport(name=’test_method_a_1_z_2’) TestCaseReport(name=’test_method_a_1_z_3’)

TestGroupReport(name=’A-2’, category=’TestSuite’)

TestCaseReport(name=’test_method_a_2_x’)

TestGroupReport(name=’B’, category=’MultiTest’)

TestGroupReport(name=’B-1’, category=’TestSuite’)

TestCaseReport(name=’test_method_b_1_x’)

TestGroupReport(name=’C’, category=’GTest’)

TestCaseReport(name=’<first test of Gtest>’) -> can only be retrieved

after GTest is run

TestCaseReport(name=’<second test of Gtest>’) -> can only be retrieved

after GTest is run

…

class testplan.report.testing.base.TestCaseReport(name, tags=None, category='testcase', **kwargs)

Bases: Report

Leaf of the report tree, contains serialized assertion / log entries.

append(item)

Append item to self.entries, no restrictions.

property counter

Return counts for current status.

classmethod deserialize(data)

Shortcut for instantiating TestCaseReport object from nested python dictionaries.

exception_logger

alias of ExceptionLogger

extend(items)

Extend self.entries with items, no restrictions.

property failed: bool

Shortcut for checking if report status should be considered failed.

filter(*functions, **kwargs)

Filtering report’s entries in place using the given functions. If any of the functions return True for a given entry, it will be kept.

flattened_entries(depth)

Need to take assertion groups into account.

property hash

Generate a hash of this report object, including its entries. This hash is used to detect when changes are made under particular nodes in the report tree. Since all report entries are mutable, this hash should NOT be used to index the report entry in a set or dict - we have avoided using the magic __hash__ method for this reason. Always use the UID for indexing purposes.

Returns:

a hash of all entries in this report group.

Return type:

int

inherit(deceased: Self) → Self

Inherit certain information from the old report, mainly for information preservation across interactive mode reloads.

is_empty() → bool

Check report is empty or not.

logged_exceptions(*exception_classes, **kwargs)

Wrapper around ExceptionRecorder, passing report arg implicitly.

Basic usage:

with report.logged_exceptions(TypeError, ValueError):
    raise some errors here ...

merge(report, strict=True)

TestCaseReport merge overwrites everything in place, as assertions of a test case won’t be split among different runners. For some special test cases, choose the one whose status is of higher precedence.

pass_if_empty()

Mark as PASSED if this testcase contains no entries.

property passed: bool

Shortcut for getting if report status should be considered passed.

reset_uid(uid=None)

Reset uid of the report, it can be useful when need to generate a global unique id instead of the current one.

property runtime_status: RuntimeStatus

Used for interactive mode, the runtime status of a testcase will be one of RuntimeStatus.

serialize()

Shortcut for serializing test report data to nested python dictionaries.

set_runtime_status_filtered(new_status: str, entries: Dict) → None

Alternative setter for the runtime status of an entry, here it is

equivalent to simply setting the runtime status.

Parameters:

• new_status – new runtime status to be set

• entries – tree-like structure of entries names, unused, but needed for current API compatibility

property status: Status

Entries in this context correspond to serialized (raw) assertions / custom logs in dictionary form. Assertion dicts will have a passed key which will be set to False for failed assertions.

property unknown: bool

Shortcut for checking if report status is unknown.

property unstable: bool

Shortcut for checking if report status should be considered unstable.

xfail(strict)

Override report status for test that is marked xfail by user :param strict: whether consider XPASS as failure

class testplan.report.testing.base.TestGroupReport(name, category='testgroup', tags=None, part=None, fix_spec_path=None, env_status=None, strict_order=False, **kwargs)

Bases: BaseReportGroup

A middle-level container report, can contain both TestGroupReports and TestCaseReports.

annotate_part_num()

append(item)

Update tag indices if item or self has tag data.

property attachments

Return all attachments from child reports.

build_index(recursive=False)

Build (refresh) indexes for this report and optionally for each child report.

This should be called explicitly if self.entries is changed.

Parameters:

recursive (bool) – Flag for triggering index build on children.

property counter: Counter

Return counts for each status, will recursively get aggregates from children and so on.

classmethod deserialize(data)

Shortcut for instantiating TestGroupReport object (and its children) from nested python dictionaries.

property entry_uids

Return the UIDs of all entries in this report group.

exception_logger

alias of ExceptionLogger

extend(items)

Add items to self.entries, checking type & index.

property failed: bool

Shortcut for checking if report status should be considered failed.

filter(*functions, **kwargs) → Self

Tag indices are updated after filter operations.

filter_by_tags(tag_value, all_tags=False) → Self

Shortcut method for filtering the report by given tags.

filter_cases(predicate: Callable[[BaseReportGroup], bool], is_root: bool = False) → Self

Case-level filter with status retained

flatten(depths=False)

Depth-first traverse the report tree starting on the leftmost node (smallest index), return a list of (depth, obj) tuples or a list of reports depending on depths flag.

Parameters:

depths – Flag for enabling/disabling depth data in result.

Returns:

List of reports or list of (depth, report) tuples.

flattened_entries(depth)

Utility function that is used by TestGroupReport.flatten.

This method should be overridden if entries have a custom hierarchy instead of a simple list.

property flattened_logs

Return a flattened list of the logs from each Report.

get_by_uid(uid: str) → Report

Get child report via uid lookup.

Parameters:

uid (hashable) – uid for the child report.

get_by_uids(uids: List[str]) → Report

Get child report via a series of uid lookup.

Parameters:

uids (list of hashable) – A uid for the child report.

graft_entry(report: Report, parent_uids: List[str])

has_uid(uid: str) → bool

Has a child report of uid

property hash

Generate a hash of this report object, including its entries. This hash is used to detect when changes are made under particular nodes in the report tree. Since all report entries are mutable, this hash should NOT be used to index the report entry in a set or dict - we have avoided using the magic __hash__ method for this reason. Always use the UID for indexing purposes.

Returns:

a hash of all entries in this report group.

Return type:

int

inherit(deceased: Self) → Self

Inherit certain information from the old report, mainly for information preservation across interactive mode reloads.

is_empty() → bool

Check report is empty or not.

logged_exceptions(*exception_classes, **kwargs)

Wrapper around ExceptionRecorder, passing report arg implicitly.

Basic usage:

with report.logged_exceptions(TypeError, ValueError):
    raise some errors here ...

merge(report, strict=True)

Propagate tag indices after merge operations.

merge_entries(report, strict=True)

For report groups, we call merge on each child report and later merge basic attributes. With default strict=True, MergeError will be raise if uid does not match. However sometimes original report may not have matching child reports. For example this happens when the test target’s test cases cannot be inspected on initialization time (e.g. GTest). In this case we can merge with strict=False so that child reports are directly appended to original report’s entries.

property passed: bool

Shortcut for getting if report status should be considered passed.

pre_order_disassemble()

pre_order_iterate()

propagate_tag_indices(parent_tags=None)

Distribute native tag data onto tags_index attributes on the nodes of the test report. This distribution happens 2 ways.

remove_by_uid(uid)

reset_uid(uid=None)

Reset uid of test report and all of its children, it can be useful when need to generate global unique id for each report entry before saving, by default strings in standard UUID format will be applied.

property runtime_status: RuntimeStatus

The runtime status is used for interactive running, and reports whether a particular entry is READY, WAITING, RUNNING, RESETTING, FINISHED or NOT_RUN.

A test group inherits its runtime status from its child entries.

serialize()

Shortcut for serializing TestGroupReport data to nested python dictionaries.

set_by_uid(uid, item)

Set child report via uid lookup.

If an entry with a matching UID is already present, that entry is updated. Otherwise a new entry will be added.

Parameters:

• uid (hashable) – uid for the child report.

• item (Report) – entry to update or insert into the report.

set_parent_uids(item)

Set the parent UIDs recursively of an item and its child entries after it has been added into this report group.

set_runtime_status_filtered(new_status: str, entries: Dict) → None

Alternative setter for the runtime status of an entry. Propagates only to the specified entries.

Parameters:

• new_status – new runtime status to be set

• entries – tree-like structure of entries names

classmethod shallow_deserialize(data, old_report)

Shortcut for deserializing a TestGroupReport object from its shallow serialized representation.

shallow_serialize()

Shortcut for shallow-serializing test report data.

property status: Status

Status of the report, will be used to decide if a Testplan run has completed successfully or not.

status_override always takes precedence, otherwise we fall back to precedent status from self.entries.

If a report group has no children, it is assumed to be passing.

property unknown: bool

Shortcut for checking if report status is unknown.

property unstable: bool

Shortcut for checking if report status should be considered unstable.

xfail(strict)

Override report status for test that is marked xfail by user :param strict: whether consider XPASS as failure

class testplan.report.testing.base.TestReport(name, meta=None, attachments=None, information=None, timeout=None, label=None, **kwargs)

Bases: BaseReportGroup

Report for a Testplan test run, sits at the root of the report tree. Only contains TestGroupReports as children.

append(item)

Add item to self.entries, checking type & index.

bubble_up_attachments()

Attachments are saved at various levels of the report:

• Fix spec file attached to multitests.

• When implemented result.attach will attach files to assertions.

This iterates through the report entries and bubbles up all the attachments to the top level. This top level dictionary of attachments will be used by Exporters to export attachments as well as the report.

build_index(recursive=False)

Build (refresh) indexes for this report and optionally for each child report.

This should be called explicitly if self.entries is changed.

Parameters:

recursive (bool) – Flag for triggering index build on children.

property counter: Counter

Return counts for each status, will recursively get aggregates from children and so on.

classmethod deserialize(data)

Shortcut for instantiating TestReport object (and its children) from nested python dictionaries.

property entry_uids

Return the UIDs of all entries in this report group.

exception_logger

alias of ExceptionLogger

extend(items)

Add items to self.entries, checking type & index.

property failed: bool

Shortcut for checking if report status should be considered failed.

filter(*functions, **kwargs) → Self

Tag indices are updated after filter operations.

filter_by_tags(tag_value, all_tags=False) → Self

Shortcut method for filtering the report by given tags.

filter_cases(predicate: Callable[[BaseReportGroup], bool], is_root: bool = False) → Self

Case-level filter with status retained

flatten(depths=False)

Depth-first traverse the report tree starting on the leftmost node (smallest index), return a list of (depth, obj) tuples or a list of reports depending on depths flag.

Parameters:

depths – Flag for enabling/disabling depth data in result.

Returns:

List of reports or list of (depth, report) tuples.

flattened_entries(depth)

Utility function that is used by TestGroupReport.flatten.

This method should be overridden if entries have a custom hierarchy instead of a simple list.

property flattened_logs

Return a flattened list of the logs from each Report.

get_by_uid(uid: str) → Report

Get child report via uid lookup.

Parameters:

uid (hashable) – uid for the child report.

get_by_uids(uids: List[str]) → Report

Get child report via a series of uid lookup.

Parameters:

uids (list of hashable) – A uid for the child report.

graft_entry(report: Report, parent_uids: List[str])

has_uid(uid: str) → bool

Has a child report of uid

property hash: int

Generate a hash of this report object, including its entries. This hash is used to detect when changes are made under particular nodes in the report tree. Since all report entries are mutable, this hash should NOT be used to index the report entry in a set or dict - we have avoided using the magic __hash__ method for this reason. Always use the UID for indexing purposes.

Returns:

a hash of all entries in this report group.

Return type:

int

inherit(deceased: Self) → Self

Inherit certain information from the old report, mainly for information preservation across interactive mode reloads.

is_empty() → bool

Check report is empty or not.

logged_exceptions(*exception_classes, **kwargs)

Wrapper around ExceptionRecorder, passing report arg implicitly.

Basic usage:

with report.logged_exceptions(TypeError, ValueError):
    raise some errors here ...

merge(report, strict=True)

Update status_override as well.

merge_entries(report, strict=True)

For report groups, we call merge on each child report and later merge basic attributes. With default strict=True, MergeError will be raise if uid does not match. However sometimes original report may not have matching child reports. For example this happens when the test target’s test cases cannot be inspected on initialization time (e.g. GTest). In this case we can merge with strict=False so that child reports are directly appended to original report’s entries.

property passed: bool

Shortcut for getting if report status should be considered passed.

pre_order_disassemble()

pre_order_iterate()

propagate_tag_indices()

TestReport does not have native tag data, so it just triggers children’s tag updates.

remove_by_uid(uid)

reset_uid(uid=None)

Reset uid of test report and all of its children, it can be useful when need to generate global unique id for each report entry before saving, by default strings in standard UUID format will be applied.

property runtime_status: RuntimeStatus

The runtime status is used for interactive running, and reports whether a particular entry is READY, WAITING, RUNNING, RESETTING, FINISHED or NOT_RUN.

A test group inherits its runtime status from its child entries.

serialize()

Shortcut for serializing test report data to nested python dictionaries.

set_by_uid(uid, item)

Set child report via uid lookup.

If an entry with a matching UID is already present, that entry is updated. Otherwise a new entry will be added.

Parameters:

• uid (hashable) – uid for the child report.

• item (Report) – entry to update or insert into the report.

set_parent_uids(item)

Set the parent UIDs recursively of an item and its child entries after it has been added into this report group.

set_runtime_status_filtered(new_status: str, entries: Dict) → None

Alternative setter for the runtime status of an entry. Propagates only to the specified entries.

Parameters:

• new_status – new runtime status to be set

• entries – tree-like structure of entries names

classmethod shallow_deserialize(data, old_report)

Shortcut for deserializing a TestReport object from its shallow serialized representation.

shallow_serialize()

Shortcut for shallow-serializing test report data.

property status: Status

Status of the report, will be used to decide if a Testplan run has completed successfully or not.

status_override always takes precedence, otherwise we fall back to precedent status from self.entries.

If a report group has no children, it is assumed to be passing.

property tags_index

Root report only has tag indexes, which is only useful when we run searches against multiple test reports. (e.g Give me all test runs from all projects that have these tags)

property unknown: bool

Shortcut for checking if report status is unknown.

property unstable: bool

Shortcut for checking if report status should be considered unstable.

xfail(strict)

Override report status for test that is marked xfail by user :param strict: whether consider XPASS as failure

testplan.report.testing.parser module

class testplan.report.testing.parser.ReportFilterAction(filter_rep: str, *args, **kwargs)

Bases: Action

Argparse action serving higher-precedence shortcuts for report filters.

static use_filter(filter_rep: str)

class testplan.report.testing.parser.ReportTagsAction(option_strings, dest, nargs=None, const=None, default=None, type=None, choices=None, required=False, help=None, metavar=None, deprecated=False)

Bases: Action

Argparse action for parsing multiple report tag arguments, builds up a list of dictionary of sets.

In:

--report-tags foo bar hello=world --report-tags one two color=red

Out:

[
    {
        'simple': {'foo', 'bar'},
        'hello': {'world'}
    },
    {
        'simple': {'one', 'two'},
        'color': {'red'},
    }
]

testplan.report.testing.schemas module

Schema classes for test Reports.

class testplan.report.testing.schemas.TestCaseReportSchema(*, only: Sequence[str] | AbstractSet[str] | None = None, exclude: Sequence[str] | AbstractSet[str] = (), many: bool | None = None, context: dict | None = None, load_only: Sequence[str] | AbstractSet[str] = (), dump_only: Sequence[str] | AbstractSet[str] = (), partial: bool | Sequence[str] | AbstractSet[str] | None = None, unknown: str | None = None)

Bases: ReportSchema

Schema for testing.TestCaseReport

dump_fields: dict[str, Field]

exclude: set[Any] | MutableSet[Any]

fields: dict[str, Field]

Dictionary mapping field_names -> Field objects

load_fields: dict[str, Field]

make_report(data, **kwargs)

Create the report object, assign timer & status_override attributes explicitly

opts: Any = <marshmallow.schema.SchemaOpts object>

source_class

alias of TestCaseReport

class testplan.report.testing.schemas.TestGroupReportSchema(*, only: Sequence[str] | AbstractSet[str] | None = None, exclude: Sequence[str] | AbstractSet[str] = (), many: bool | None = None, context: dict | None = None, load_only: Sequence[str] | AbstractSet[str] = (), dump_only: Sequence[str] | AbstractSet[str] = (), partial: bool | Sequence[str] | AbstractSet[str] | None = None, unknown: str | None = None)

Bases: BaseReportGroupSchema

Schema for testing.TestGroupReportSchema, supports tree serialization.

dump_fields: dict[str, Field]

exclude: set[Any] | MutableSet[Any]

fields: dict[str, Field]

Dictionary mapping field_names -> Field objects

load_fields: dict[str, Field]

make_report(data, **kwargs)

Propagate tag indices after deserialization

opts: Any = <marshmallow.schema.SchemaOpts object>

source_class

alias of TestGroupReport

class testplan.report.testing.schemas.TestReportSchema(*, only: Sequence[str] | AbstractSet[str] | None = None, exclude: Sequence[str] | AbstractSet[str] = (), many: bool | None = None, context: dict | None = None, load_only: Sequence[str] | AbstractSet[str] = (), dump_only: Sequence[str] | AbstractSet[str] = (), partial: bool | Sequence[str] | AbstractSet[str] | None = None, unknown: str | None = None)

Bases: BaseReportGroupSchema

Schema for test report root, testing.TestReport.

class Meta

Bases: object

unknown = 'exclude'

dump_fields: dict[str, Field]

exclude: set[Any] | MutableSet[Any]

fields: dict[str, Field]

Dictionary mapping field_names -> Field objects

load_fields: dict[str, Field]

make_report(data, **kwargs)

Create report object & deserialize sub trees.

opts: Any = <marshmallow.schema.SchemaOpts object>

source_class

alias of TestReport

testplan.report.testing.styles module

Styling enums & flags for rendering test output.

Test report outputs will use common styling flags to render the test data on different outputs (e.g. console, pdf, web), using custom information levels (multitests, suites, tescases, assrtions, assertion details).

We also allow separate styling for passing and failing tests, so it is possible to get assertion details for failing tests but just return high level multitest pass/fail status for passing test groups.

class testplan.report.testing.styles.Style(passing, failing)

Bases: object

Container for StyleFlag objects, we will make use of 2 StyleFlag objects for passing / failing test result rendering.

e.g. Render passing multitests, but render failing multitest & suite & testcases.

get_style(passing=True)

class testplan.report.testing.styles.StyleArg(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: ArgMixin, Enum

Argparse utility that gives us StyleFlag tuples for matching argument values. This will be used for shortcut styling of outputs (e.g. stdout, pdf) from cmdline args.

For fine tuning how the output is rendered, the style object must be passed to related renderer programmatically.

DETAILED = Style(passing='assertion-detail', failing='assertion-detail')

EXTENDED_SUMMARY = Style(passing='testcase', failing='assertion-detail')

RESULT_ONLY = Style(passing='result', failing='result')

SUMMARY = Style(passing='test', failing='test')

classmethod get_descriptions()

Description mapping for enums, will be rendered via –help command.

class testplan.report.testing.styles.StyleEnum(value, names=<not given>, *values, module=None, qualname=None, type=None, start=1, boundary=None)

Bases: ArgMixin, Enum

Incremental output levels, this will be used by StyleFlag class to set styling flags like:

• display_result = True

• display_multitest = False

Note: Multiple inheritance seems to break with IntEnum.

ASSERTION = 4

ASSERTION_DETAIL = 5

RESULT = 0

TEST = 1

TESTCASE = 3

TESTSUITE = 2

class testplan.report.testing.styles.StyleFlag(level)

Bases: object

This class generates styling attributes using the given output level. It will use the incremental values of StyleEnum to set its display flag attributes.

This gives us a nice flag with clear attributes to work with when we implement the rendering logic.

Usage:

StyleFlag(StyleEnum.TESTCASE)

Alternative Usage:

StyleFlag(‘testcase’)

Produces:

StyleFlag

• display_result = True

• display_test = True

• display_testsuite = True

• display_testcase = True

• display_assertion = False

• display_assertion_detail = False

attrs = ['display_result', 'display_test', 'display_testsuite', 'display_testcase', 'display_assertion', 'display_assertion_detail']

property level

Need this as a property rather than attribute, so we can pickle StyleFlag objects

Module contents

Report classes for Testplan