Output

Testplan provides comprehensive test output styles made available from the built-in test report exporters. These are the following:

Console

Based on the verbosity specified with command line options or programmatically (see the downloadable example that demonstrates that), the assertions/testcases/testsuites/Multitest level results will be printed on the console:

• Basic assertions verbose output:

  

• Table assertions verbose output:

  

• Fix/Dict match assertions verbose output:

  

PDF

PDF output concept is similar to console but it is used when persistent test evidence are needed. PDF reports can be generated by either command line options, or be done programmatically with configuration options of Testplan.

Examples for PDF report generation can be seen here.

• Basic assertions PDF representation:

  

• Table assertions PDF representation:

  

• Fix/Dict match assertions PDF representation:

  

• Matplot assertions PDF representation:

  

Testplan uses Reportlab to generate test reports in PDF format. The simplest way to enable this functionality is to pass --pdf comand line argument:

$ ./test_plan.py --pdf my-report.pdf
    [MultiTest] -> Pass
[Testplan] -> Pass
PDF generated at mypdf.pdf

It is also possible to use programmatic declaration for PDF report generation as well:

@test_plan(name='SamplePlan', pdf_path='my-report.pdf')
def main(plan):
    # Testplan implicitly creates PDF exporter if we just pass `pdf_path`
    ...

A more explicit usage is to initialize a PDF exporter directly:

from testplan.exporters.testing import PDFExporter

@test_plan(
  name='SamplePlan',
  exporters=[
      PDFExporter(pdf_path='my-report.pdf')
  ]
)
def main(plan):
    ...

PDF reports can contain different levels of detail, configured via styling options. These options can be specified:

• via command line:

  $ ./test_plan.py --pdf my-report.pdf --pdf-style extended-summary
  

• programmatically:

  from testplan.report.testing.styles import Style, StyleEnum
  
  @test_plan(
      name='SamplePlan',
      pdf_path='my-report.pdf',
      pdf_style=Style(
          passing=StyleEnum.ASSERTION,
          failing=StyleEnum.ASSERTION_DETAIL
      )
  )
  def main(plan):
      ...
  

Read more about output styles here.

Tag filtered PDFs

If a plan has a very large number of tests, it may be better to generate multiple PDF reports (filtered by tags), rather a single report.

Testplan provides such functionality via tag filtered PDF generation, which can be enabled by --report-tags and --report-tags-all arguments:

Example tagger testsuite and testcase:

@testsuite(tags='alpha')
class SampleTestAlpha(object):

    @testcase(tags='server')
    def test_method_1(self, env, result):
        ...

    @testcase(tags='database')
    def test_method_2(self, env, result):
        ...

    @testcase(tags=('server', 'client'))
    def test_method_3(self, env, result):
        ...

The command below will generate 2 PDFs, first one will contain test results from tests tagged with database, second one will contain the results from tests tagged with server OR client

A new PDF will be generated for each --report-tags/--report-tags-all argument.

$ ./test_plan.py --report-dir ./reports --report-tags database --report-tags server client

Equivalent programmatic declaration for the same reports would be:

@test_plan(
  name='SamplePlan',
  report_dir='reports'
  report_tags=[
      'database',
      ('server', 'client')
  ]
)
def main(plan):
    # Testplan implicitly creates Tag Filtered PDF exporter if we pass
    # the `report_tags` / `report_tags_all` arguments.
    ...

A more explicit usage is to initialize a Tag Filtered PDF exporter directly:

from testplan.exporters.testing import TagFilteredPDFExporter

@test_plan(
    name='SamplePlan',
    exporters=[
        TagFiltered(
            report_dir='reports',
            report_tags=[
                'database',
                ('server', 'client')
            ]
        )
    ]
)
def main(plan):
   ...

Examples for Tag filtered PDF report generation can be seen here.

XML

XML output can be generated per each MultiTest in a plan, and can be used as an alternative format for persistent test evidence. Testplan supports XML exports compatible with the JUnit format:

<testsuites>
  <testsuite tests="3" errors="0" name="AlphaSuite" package="Primary:AlphaSuite" hostname="hostname.example.com" failures="1" id="0">
    <testcase classname="Primary:AlphaSuite:test_equality_passing" name="test_equality_passing" time="0.138505"/>
    <testcase classname="Primary:AlphaSuite:test_equality_failing" name="test_equality_failing" time="0.001906">
      <failure message="failing equality" type="assertion"/>
    </testcase>
    <testcase classname="Primary:AlphaSuite:test_membership_passing" name="test_membership_passing" time="0.00184"/>
    <system-out/>
    <system-err/>
  </testsuite>
</testsuites>

To enable this functionality , use --xml and provide a directory for XML files. It is possible to generate an XML file per each MultiTest in your plan.

$ ./test_plan.py --xml /path/to/xml-dir

It is also possible to use programmatic declaration for XML generation as well:

@test_plan(name='SamplePlan', xml_dir='/path/to/xml-dir')
def main(plan):
    # Testplan implicitly creates XML exporter if we just pass `xml_dir`
    ...

A more explicit usage is to initialize a XML exporter directly.

from testplan.exporters.testing import XMLExporter

@test_plan(
    name='SamplePlan',
    exporters=[
        XMLExporter(xml_dir='/path/to/xml-dir')
    ]
)
def main(plan):
    ...

Examples for XML report generation can be seen here.

JSON

A JSON file fully represents the test data. Testplan supports serialization / deserialization of test reports, meaning that report can be stored as a JSON file and then loaded back into memory to generate other kinds of output (e.g. PDF, XML or any custom export target).

Sample JSON output:

{"entries": [{
  "category": "multitest",
  "description": null,
  "entries": [{
    "category": "suite",
    "description": null,
    "entries": [{
      "description": null,
      "entries": [{
        "category": null,
        "description": "passing equality",
        "first": 1,
        "label": "==",
        "line_no": 54,
        "machine_time": "2018-02-05T15:16:40.951528+00:00",
        "meta_type": "assertion",
        "passed": True,
        "second": 1,
        "type": "Equal",
        "utc_time": "2018-02-05T15:16:40.951516+00:00"}
      ],
      "logs": [],
      "name": "passing_testcase_one",
      "status": "passed",
      "status_override": null,
      "tags": {},
      "tags_index": {},
      "timer": {
        "run": {
          "end": "2018-02-05T15:16:41.164086+00:00",
          "start": "2018-02-05T15:16:40.951456+00:00"}},
          "type": "TestCaseReport",
          "uid": "9b4467e2-668c-4764-942b-061ea58da0f0"
        },
      ...
      ],
      "logs": [],
      "name": "BetaSuite",
      "status": "passed",
      "status_override": null,
      "tags": {},
      "tags_index": {},
      "timer": {},
      "type": "TestGroupReport",
      "uid": "eeb87e19-ffcb-4710-8eeb-6daff89c46d9"}],
      "logs": [],
      "name": "MyMultitest",
      "status": "passed",
      "status_override": null,
      "tags": {},
      "tags_index": {},
      "timer": {},
      "type": "TestGroupReport",
      "uid": "bf44e942-c267-42b9-b379-5ec8c4c7878b"}
    ],
   "meta": {},
   "name": "Basic JSON Report Example",
   "status": "passed",
   "status_override": null,
   "tags_index": {},
   "timer": {
    "run": {
      "end": "2018-02-05T15:16:41.188904+00:00",
      "start": "2018-02-05T15:16:40.937402+00:00"
    }
  },
  uid": "5d541277-e0c4-43c6-941b-dea2c7d3259c"
}

A JSON report can be generated via --json argument:

$ ./test_plan.py --json /path/to/json

Same result can be achieved by programmatic declaration as well:

@test_plan(name='Sample Plan', json_path='/path/to/json')
def main(plan):
    # Testplan implicitly creates JSON exporter if we just pass `json_path`
    ...

A more explicit usage is to initialize a JSON exporter directly:

from testplan.exporters.testing import JSONExporter

@test_plan(
    name='Sample Plan',
    exporters=[
        JSONExporter(json_path='/path/to/json')
    ]
)
def main(plan):
    ...

Examples for JSON report generation can be seen here.

Browser

Testplan can store the report locally as a JSON file and then starts a web server to load and parse it, thus it is convenient for user to open the test report in the browser such as Chrome or Firefox, like below:

• Basic assertions Browser UI representation:

  

• Table assertions Browser UI representation:

  

• Fix/Dict match assertions Browser UI represenation:

  

Command line option --ui can be used to start a local web server after Testplan runs. A port number can be specified after the argument, if ignored a random port will be used. Testplan will print a URL to console which can be opened in a browser to view the report.

$ ./test_plan.py --ui 12345

Or programmatically declare it:

@test_plan(name='SamplePlan', ui_port=12345)
def main(plan):
    # Testplan implicitly starts a web browser and use can browse the test report
    ...

This feature requires the install-testplan-ui script to have been run (see Install Testplan in the Getting Started section). If this script hasn’t been run the web server will start but the report won’t load in the browser.

An alternative is to initialize a web server exporter directly. But it is recommended to place this exporter last in the list after all other exporters. Because this exporter will cause Testplan to block, while other exporters might also have post exporter steps to be completed.

from testplan.exporters.testing import WebServerExporter

@test_plan(
    name='Sample Plan',
    exporters=[
        WebServerExporter(ui_port=12345)
    ]
)
def main(plan):
    ...