testplan.common.report package

Submodules

testplan.common.report.base module

Base classes for building a report tree.

The idea behind this module is that we may have parallel runners for a testplan, each of which would generate a partial report.

Later on these reports would be merged together to build the final report as the testplan result.

class testplan.common.report.base.BaseReportGroup(name, **kwargs)

Bases: testplan.common.report.base.Report

A report class that contains other Reports or ReportGroups in entries. Allows O(1) child report lookup via get_by_uid method.

append(item)

Add item to self.entries, checking type & index.

build_index(recursive=False)

Build (refresh) indexes for this report and optionally for each child report.

This should be called explicitly if self.entries is changed.

Parameters:recursive (bool) – Flag for triggering index build on children.

counter

Return counts for each status, will recursively get aggregates from children and so on.

entry_uids

Return the UIDs of all entries in this report group.

exception_logger

alias of ExceptionLogger

extend(items)

Add items to self.entries, checking type & index.

failed

Shortcut for checking if report status should be considered failed.

filter(*functions, **kwargs) → typing_extensions.Self

Recursively filter report entries and sub-entries.

filter_by_tags(tag_value, all_tags=False) → typing_extensions.Self

Shortcut method for filtering the report by given tags.

filter_cases(predicate: Callable[[BaseReportGroup], bool], is_root: bool = False) → typing_extensions.Self

Case-level filter with status retained

flatten(depths=False)

Depth-first traverse the report tree starting on the leftmost node (smallest index), return a list of (depth, obj) tuples or a list of reports depending on depths flag.

Parameters:depths – Flag for enabling/disabling depth data in result. Returns:List of reports or list of (depth, report) tuples.

flattened_logs

Return a flattened list of the logs from each Report.

get_by_uid(uid)

Get child report via uid lookup.

Parameters:uid (hashable) – uid for the child report.

get_by_uids(uids)

Get child report via a series of uid lookup.

Parameters:uids (list of hashable) – A uid for the child report.

has_uid(uid)

Has a child report of uid

hash

Generate a hash of this report object, including its entries. This hash is used to detect when changes are made under particular nodes in the report tree. Since all report entries are mutable, this hash should NOT be used to index the report entry in a set or dict - we have avoided using the magic __hash__ method for this reason. Always use the UID for indexing purposes.

Returns:a hash of all entries in this report group. Return type:int

merge(report, strict=True)

Update status_override as well.

merge_entries(report, strict=True)

For report groups, we call merge on each child report and later merge basic attributes. With default strict=True, MergeError will be raise if uid does not match. However sometimes original report may not have matching child reports. For example this happens when the test target’s test cases cannot be inspected on initialization time (e.g. GTest). In this case we can merge with strict=False so that child reports are directly appended to original report’s entries.

passed

Shortcut for getting if report status should be considered passed.

remove_by_uid(uid)

reset_uid(uid=None)

Reset uid of test report and all of its children, it can be useful when need to generate global unique id for each report entry before saving, by default strings in standard UUID format will be applied.

runtime_status

The runtime status is used for interactive running, and reports whether a particular entry is READY, WAITING, RUNNING, RESETTING, FINISHED or NOT_RUN.

A test group inherits its runtime status from its child entries.

set_by_uid(uid, item)

Set child report via uid lookup.

If an entry with a matching UID is already present, that entry is updated. Otherwise a new entry will be added.

Parameters:

• uid (hashable) – uid for the child report.
• item (Report) – entry to update or insert into the report.

set_parent_uids(item)

Set the parent UIDs recursively of an item and its child entries after it has been added into this report group.

set_runtime_status_filtered(new_status: str, entries: Dict[KT, VT]) → None

Alternative setter for the runtime status of an entry. Propagates only

to the specified entries.

Parameters:

• new_status – new runtime status to be set
• entries – tree-like structure of entries names

status

Status of the report, will be used to decide if a Testplan run has completed successfully or not.

status_override always takes precedence, otherwise we fall back to precedent status from self.entries.

If a report group has no children, it is assumed to be passing.

unknown

Shortcut for checking if report status is unknown.

unstable

Shortcut for checking if report status should be considered unstable.

xfail(strict)

Override report status for test that is marked xfail by user :param strict: whether consider XPASS as failure

class testplan.common.report.base.ExceptionLogger(*exception_classes, **kwargs)

Bases: testplan.common.report.base.ExceptionLoggerBase

When we run tests, we always want to return a report object, However we also want to mark the test as failed if an exception is raised (unless kwargs[‘fail’] is False).

class testplan.common.report.base.ExceptionLoggerBase(*exception_classes, **kwargs)

Bases: object

A context manager used for suppressing & logging an exception.

exception testplan.common.report.base.MergeError

Bases: Exception

Raised when a merge operation fails.

class testplan.common.report.base.Report(name: str, description: Optional[str] = None, definition_name: Optional[str] = None, uid: Optional[str] = None, entries: Optional[list] = None, parent_uids: Optional[List[str]] = None, status_override=None, status_reason=None)

Bases: object

Base report class, support exception suppression & logging via report.logged_exceptions. Stores arbitrary objects in entries attribute.

append(item)

Append item to self.entries, no restrictions.

exception_logger

alias of ExceptionLoggerBase

extend(items)

Extend self.entries with items, no restrictions.

filter(*functions, **kwargs)

Filtering report’s entries in place using the given functions. If any of the functions return True for a given entry, it will be kept.

flattened_entries(depth)

Utility function that is used by TestGroupReport.flatten.

This method should be overridden if entries have a custom hierarchy instead of a simple list.

hash

Return a hash of all entries in this report.

inherit(deceased: typing_extensions.Self) → typing_extensions.Self

Inherit certain information from the old report, mainly for information preservation across interactive mode reloads.

is_empty() → bool

Check report is empty or not.

logged_exceptions(*exception_classes, **kwargs)

Wrapper around ExceptionRecorder, passing report arg implicitly.

Basic usage:

with report.logged_exceptions(TypeError, ValueError):
    raise some errors here ...

merge(report, strict=True)

Child classes can override this, just make sure merge operation is idempotent, so that merging the same report onto self multiple times does not create extra data.

Parameters:

• report (Report) – Report that contains logs to merge.
• strict (bool) – Flag for enabling / disabling strict merge ops.

reset_uid(uid=None)

Reset uid of the report, it can be useful when need to generate a global unique id instead of the current one.

class testplan.common.report.base.ReportCategories

Bases: str, enum.Enum

Enumeration of possible categories of report nodes.

Note: we don’t use the enum.Enum base class to simplify report serialization via marshmallow.

BOOST_SUITE = 'boost-suite'

BOOST_TEST = 'boost-test'

CPPUNIT = 'cppunit'

CPPUNIT_SUITE = 'cppunit-suite'

ERROR = 'error'

GTEST = 'gtest'

GTEST_SUITE = 'gtest-suite'

HOBBESTEST = 'hobbestest'

HOBBESTEST_SUITE = 'hobbestest-suite'

JUNIT = 'junit'

MULTITEST = 'multitest'

PARAMETRIZATION = 'parametrization'

PYTEST = 'pytest'

PYUNIT = 'pyunit'

QUNIT = 'qunit'

SYNTHESIZED = 'synthesized'

TASK_RERUN = 'task_rerun'

TESTCASE = 'testcase'

TESTGROUP = 'testgroup'

TESTPLAN = 'testplan'

TESTSUITE = 'testsuite'

UNITTEST = 'unittest'

class testplan.common.report.base.RuntimeStatus

Bases: enum.Enum

Constants for test runtime status - for interactive mode

total order encoded in value, serialized to “lower-case string” of name

FINISHED = 6

NONE = 7

NOT_RUN = 5

READY = 4

RESETTING = 2

RUNNING = 1

WAITING = 3

from_json_compatible = <bound method RuntimeStatus.from_json_compatible of <enum 'RuntimeStatus'>>

precede(other: typing_extensions.Self) → bool

Return self<value.

precedent = <bound method RuntimeStatus.precedent of <enum 'RuntimeStatus'>>

to_json_compatible() → Optional[str]

exception testplan.common.report.base.SkipTestcaseException

Bases: Exception

Raised from an explicit call to result.skip.

class testplan.common.report.base.Status

Bases: enum.Enum

Constants for test result and utilities for propagating status upward.

partial order encoded by value, serialized to “lower-case string” of name

tens of value encoding status category, tenths of value only for differentiating enum members

BOTTOM = -1

ERROR = 9

FAILED = 19

INCOMPLETE = 18.1

NONE = 59

PASSED = 39

SKIPPED = 48.1

UNKNOWN = 29

UNSTABLE = 49

XFAIL = 48.2

XPASS = 48.3

XPASS_STRICT = 18.2

from_json_compatible = <bound method Status.from_json_compatible of <enum 'Status'>>

normalised() → typing_extensions.Self

precede(other: typing_extensions.Self) → bool

precedent = <bound method Status.precedent of <enum 'Status'>>

to_json_compatible() → Optional[str]

testplan.common.report.log module

We’d like to use python’s logging interface when we log messages through Report objects. The most lightweight way to do it is to use a logging.LoggingAdapter which is a thin wrapper around a logging.Logger.

We can then use a global mapping and a custom handler to append log messages to the report object’s logs list.

class testplan.common.report.log.ReportLogHandler(level=0)

Bases: logging.Handler

Log handler that uses the global report map for appending log messages to report objects.

emit(record)

Do whatever it takes to actually log the specified logging record.

This version is intended to be implemented by subclasses and so raises a NotImplementedError.

testplan.common.report.log.create_logging_adapter(report)

Create a new adapter and bind the report to global REPORT_MAP so handler can access it.

testplan.common.report.schemas module

Base schemas for report serialization.

class testplan.common.report.schemas.ReportLogSchema(*, only: types.StrSequenceOrSet | None = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: dict | None = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: bool | types.StrSequenceOrSet = False, unknown: str | None = None)

Bases: marshmallow.schema.Schema

Schema for log record data created by report stdout.

opts = <marshmallow.schema.SchemaOpts object>

class testplan.common.report.schemas.ReportSchema(*, only: types.StrSequenceOrSet | None = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: dict | None = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: bool | types.StrSequenceOrSet = False, unknown: str | None = None)

Bases: testplan.common.serialization.schemas.TreeNodeSchema

Schema for base.Report.

class Meta

Bases: object

unknown = 'exclude'

make_report(data, **kwargs)

Create report object, attach log list.

opts = <marshmallow.schema.SchemaOpts object>

source_class

alias of testplan.common.report.base.Report

class testplan.common.report.schemas.BaseReportGroupSchema(*, only: types.StrSequenceOrSet | None = None, exclude: types.StrSequenceOrSet = (), many: bool = False, context: dict | None = None, load_only: types.StrSequenceOrSet = (), dump_only: types.StrSequenceOrSet = (), partial: bool | types.StrSequenceOrSet = False, unknown: str | None = None)

Bases: testplan.common.report.schemas.ReportSchema

Schema for base.BaseReportGroup.

make_report(data, **kwargs)

Create report group object

opts = <marshmallow.schema.SchemaOpts object>

source_class

alias of testplan.common.report.base.BaseReportGroup

Module contents