Output
******

Testplan provides comprehensive test output styles made available from
the built-in test report exporters. These are the following:

.. _Output_Console:

Console
=======

Based on the verbosity specified with command line options or programmatically
(see the :ref:`downloadable example <example_test_output_console>` that demonstrates that),
the assertions/testcases/testsuites/Multitest level results will be printed on the console:

    * **Basic** assertions verbose output:

        .. image:: ../images/output/console/basic_assertions.png

    * **Table** assertions verbose output:

        .. image:: ../images/output/console/table_assertions.png

    * **Fix/Dict** match assertions verbose output:

        .. image:: ../images/output/console/fix_assertions.png


.. _Output_PDF:

PDF
===

PDF output concept is similar to console but it is used when persistent test evidence
are needed. PDF reports can be generated by either command line options, or be done
programmatically with configuration options of Testplan.

Examples for PDF report generation can be seen :ref:`here <example_test_output_exporters_pdf>`.

    * **Basic** assertions PDF representation:

        .. image:: ../images/output/pdf/basic_assertions.png

    * **Table** assertions PDF representation:

        .. image:: ../images/output/pdf/table_assertions.png

    * **Fix/Dict** match assertions PDF representation:

        .. image:: ../images/output/pdf/fix_assertions.png

    * **Matplot** assertions PDF representation:

        .. image:: ../images/output/pdf/matplot_assertions.png

Testplan uses `Reportlab <http://www.reportlab.com/opensource/>`_ to generate
test reports in PDF format. The simplest way to enable this functionality is to
pass ``--pdf`` comand line argument:

.. code-block:: bash

    $ ./test_plan.py --pdf my-report.pdf
        [MultiTest] -> Pass
    [Testplan] -> Pass
    PDF generated at mypdf.pdf


It is also possible to use programmatic declaration for PDF report generation
as well:

.. code-block:: python

    @test_plan(name='SamplePlan', pdf_path='my-report.pdf')
    def main(plan):
        # Testplan implicitly creates PDF exporter if we just pass `pdf_path`
        ...

A more explicit usage is to initialize a PDF exporter directly:

.. code-block:: python

    from testplan.exporters.testing import PDFExporter

    @test_plan(
      name='SamplePlan',
      exporters=[
          PDFExporter(pdf_path='my-report.pdf')
      ]
    )
    def main(plan):
        ...

PDF reports can contain different levels of detail, configured via styling
options. These options can be specified:

    * via command line:

        .. code-block:: bash

            $ ./test_plan.py --pdf my-report.pdf --pdf-style extended-summary

    * programmatically:

        .. code-block:: python

            from testplan.report.testing.styles import Style, StyleEnum

            @test_plan(
                name='SamplePlan',
                pdf_path='my-report.pdf',
                pdf_style=Style(
                    passing=StyleEnum.ASSERTION,
                    failing=StyleEnum.ASSERTION_DETAIL
                )
            )
            def main(plan):
                ...

Read more about output styles :ref:`here <styling_output>`.


.. _Output_Tag_Filtered_PDF:

Tag filtered PDFs
=================

If a plan has a very large number of tests, it may be better to generate
multiple PDF reports (filtered by tags), rather a single report.

Testplan provides such functionality via tag filtered PDF generation, which can
be enabled by ``--report-tags`` and ``--report-tags-all`` arguments:

Example tagger testsuite and testcase:

.. code-block:: python

    @testsuite(tags='alpha')
    class SampleTestAlpha(object):

        @testcase(tags='server')
        def test_method_1(self, env, result):
            ...

        @testcase(tags='database')
        def test_method_2(self, env, result):
            ...

        @testcase(tags=('server', 'client'))
        def test_method_3(self, env, result):
            ...

The command below will generate 2 PDFs, first one will contain test results from
tests tagged with ``database``, second one will contain the results from tests
tagged with ``server`` OR ``client``

A new PDF will be generated for each ``--report-tags``/``--report-tags-all``
argument.

.. code-block:: bash

    $ ./test_plan.py --report-dir ./reports --report-tags database --report-tags server client

Equivalent programmatic declaration for the same reports would be:

.. code-block:: python

    @test_plan(
      name='SamplePlan',
      report_dir='reports'
      report_tags=[
          'database',
          ('server', 'client')
      ]
    )
    def main(plan):
        # Testplan implicitly creates Tag Filtered PDF exporter if we pass
        # the `report_tags` / `report_tags_all` arguments.
        ...

A more explicit usage is to initialize a Tag Filtered PDF exporter directly:

.. code-block:: python

    from testplan.exporters.testing import TagFilteredPDFExporter

    @test_plan(
        name='SamplePlan',
        exporters=[
            TagFiltered(
                report_dir='reports',
                report_tags=[
                    'database',
                    ('server', 'client')
                ]
            )
        ]
    )
    def main(plan):
       ...

Examples for Tag filtered PDF report generation can be seen :ref:`here <example_test_output_exporters_pdf>`.


.. _Output_XML:

XML
===

XML output can be generated per each MultiTest in a plan, and can be used as an
alternative format for persistent test evidence. Testplan supports XML exports
compatible with the JUnit format:

.. code-block:: xml

    <testsuites>
      <testsuite tests="3" errors="0" name="AlphaSuite" package="Primary:AlphaSuite" hostname="hostname.example.com" failures="1" id="0">
        <testcase classname="Primary:AlphaSuite:test_equality_passing" name="test_equality_passing" time="0.138505"/>
        <testcase classname="Primary:AlphaSuite:test_equality_failing" name="test_equality_failing" time="0.001906">
          <failure message="failing equality" type="assertion"/>
        </testcase>
        <testcase classname="Primary:AlphaSuite:test_membership_passing" name="test_membership_passing" time="0.00184"/>
        <system-out/>
        <system-err/>
      </testsuite>
    </testsuites>

To enable this functionality , use ``--xml`` and provide a directory for XML files.
It is possible to generate an XML file per each MultiTest in your plan.

.. code-block:: bash

    $ ./test_plan.py --xml /path/to/xml-dir

It is also possible to use programmatic declaration for XML generation as well:

.. code-block:: python

    @test_plan(name='SamplePlan', xml_dir='/path/to/xml-dir')
    def main(plan):
        # Testplan implicitly creates XML exporter if we just pass `xml_dir`
        ...

A more explicit usage is to initialize a XML exporter directly.

.. code-block:: python

    from testplan.exporters.testing import XMLExporter

    @test_plan(
        name='SamplePlan',
        exporters=[
            XMLExporter(xml_dir='/path/to/xml-dir')
        ]
    )
    def main(plan):
        ...

Examples for XML report generation can be seen :ref:`here <example_test_output_exporters_xml>`.


.. _Output_JSON:

JSON
====

A JSON file fully represents the test data. Testplan supports serialization / deserialization
of test reports, meaning that report can be stored as a JSON file and then loaded back into
memory to generate other kinds of output (e.g. PDF, XML or any custom export target).

Sample JSON output:

.. code-block:: python

  {"entries": [{
    "category": "multitest",
    "description": null,
    "entries": [{
      "category": "suite",
      "description": null,
      "entries": [{
        "description": null,
        "entries": [{
          "category": null,
          "description": "passing equality",
          "first": 1,
          "label": "==",
          "line_no": 54,
          "machine_time": "2018-02-05T15:16:40.951528+00:00",
          "meta_type": "assertion",
          "passed": True,
          "second": 1,
          "type": "Equal",
          "utc_time": "2018-02-05T15:16:40.951516+00:00"}
        ],
        "logs": [],
        "name": "passing_testcase_one",
        "status": "passed",
        "status_override": null,
        "tags": {},
        "tags_index": {},
        "timer": {
          "run": {
            "end": "2018-02-05T15:16:41.164086+00:00",
            "start": "2018-02-05T15:16:40.951456+00:00"}},
            "type": "TestCaseReport",
            "uid": "9b4467e2-668c-4764-942b-061ea58da0f0"
          },
        ...
        ],
        "logs": [],
        "name": "BetaSuite",
        "status": "passed",
        "status_override": null,
        "tags": {},
        "tags_index": {},
        "timer": {},
        "type": "TestGroupReport",
        "uid": "eeb87e19-ffcb-4710-8eeb-6daff89c46d9"}],
        "logs": [],
        "name": "MyMultitest",
        "status": "passed",
        "status_override": null,
        "tags": {},
        "tags_index": {},
        "timer": {},
        "type": "TestGroupReport",
        "uid": "bf44e942-c267-42b9-b379-5ec8c4c7878b"}
      ],
     "meta": {},
     "name": "Basic JSON Report Example",
     "status": "passed",
     "status_override": null,
     "tags_index": {},
     "timer": {
      "run": {
        "end": "2018-02-05T15:16:41.188904+00:00",
        "start": "2018-02-05T15:16:40.937402+00:00"
      }
    },
    uid": "5d541277-e0c4-43c6-941b-dea2c7d3259c"
  }

A JSON report can be generated via ``--json`` argument:

.. code-block:: bash

  $ ./test_plan.py --json /path/to/json

Same result can be achieved by programmatic declaration as well:

.. code-block:: python

    @test_plan(name='Sample Plan', json_path='/path/to/json')
    def main(plan):
        # Testplan implicitly creates JSON exporter if we just pass `json_path`
        ...

A more explicit usage is to initialize a JSON exporter directly:

.. code-block:: python

    from testplan.exporters.testing import JSONExporter

    @test_plan(
        name='Sample Plan',
        exporters=[
            JSONExporter(json_path='/path/to/json')
        ]
    )
    def main(plan):
        ...

Examples for JSON report generation can be seen :ref:`here <example_test_output_exporters_json>`.


.. _Output_Browser:

Browser
=======

Testplan can store the report locally as a JSON file and then starts a web server
to load and parse it, thus it is convenient for user to open the test report
in the browser such as Chrome or Firefox, like below:

    * **Basic** assertions Browser UI representation:

        .. image:: ../images/output/browser/basic_assertions.png

    * **Table** assertions Browser UI representation:

        .. image:: ../images/output/browser/table_assertions.png

    * **Fix/Dict** match assertions Browser UI represenation:

        .. image:: ../images/output/browser/fix_assertions.png

Command line option ``--ui`` can be used to start a local web server after Testplan runs.
A port number can be specified after the argument, if ignored a random port will be used.
Testplan will print a URL to console which can be opened in a browser to view the report.

.. code-block:: bash

    $ ./test_plan.py --ui 12345

Or programmatically declare it:

.. code-block:: python

    @test_plan(name='SamplePlan', ui_port=12345)
    def main(plan):
        # Testplan implicitly starts a web browser and use can browse the test report
        ...

This feature requires the ``install-testplan-ui`` script to have been run (see
:ref:`Install Testplan <install_testplan>` in the Getting Started section). If
this script hasn't been run the web server will start but the report won't load
in the browser.

An alternative is to initialize a web server exporter directly. But it is recommended
to place this exporter last in the list after all other exporters. Because this exporter
will cause Testplan to block, while other exporters might also have post exporter steps
to be completed.

.. code-block:: python

    from testplan.exporters.testing import WebServerExporter

    @test_plan(
        name='Sample Plan',
        exporters=[
            WebServerExporter(ui_port=12345)
        ]
    )
    def main(plan):
        ...