BDD

testplan.testing.bdd package makes it possible to use Gherkin to express testcases. Testplan can execute test suites expressed in feature files.

Quick Start

Create the following files in your project, or get started using the downloadable example.

features/first.feature

features/steps/steps.py

test_plan.py

Now you should have a project structure like below:

> tree
.
|-- features
|   |-- first.feature
|   `-- steps
|       `-- steps.py
`-- test_plan.py

Give it a go

./test_plan.py

You should see something like this:

        [Given we have two number: 1 and 1] -> Passed
        [When we sum the numbers] -> Passed
        [Then the result is: 2] -> Passed
      [1 + 1] -> Passed
        [Given we have two number: 2 and 2] -> Passed
        [When we sum the numbers] -> Passed
        [Then the result is: 4] -> Passed
      [2 + 2] -> Passed
    [Example Gherkin Testsuite] -> Passed
  [Example Gherkin Test] -> Passed
[Example Gherkin Testplan] -> Passed

Basics

As in most Gherkin based BDD frameworks, the English written sentences, Given/When/Then …, need to be turned into code. We call the sentences as steps, and the code they trigger as step definitions. To match steps with their definition, there are certain rules on directory layout and/or naming. We use code tree layout instead of explicitly setting connection in code, to reduce the boilerplate every Testplan need to have.

The directory structure should look like:

tree
.
|-- features
|   |-- first.feature
|   |-- second.feature
|   |-- important
|   |   `-- another_feature
|   `-- steps
|       |-- first_steps.py
|       |-- second_steps.py
|       `-- common_steps.py
`-- testplan.py

The feature files can be organized into folders, though the structure does not have any meaning in the test execution. There can be as many step definition files as needed.

Using the factory

In current state of the BDD framework, it provide a factory that can transform feature files and step definitions into proper Multitest testsuites.

BDDTestSuiteFactory(features_path, resolver=NoopContextResolver(), default_parser=RegExParser)

• features_path: is the path to the directory containing the feature and step definition
• resolver: there is a simple way to refer data from context. See []()
• default_parser: see [Parsers](#parsers) (SimpleParser or RegExParser)
• feature_linked_steps: see [Step Definitions](#step-definitions)

Standard Gherkin Features

This Gherkin Reference should help to start with Gherkin, but the important standard features, and how they translate to Testplan will be discussed here.

Features and Scenarios

Features maped to Testsuites and Scenarios are maped to Testcases, having this mapping enable to nicely mix BDD style with Testplan style within the same test.

Scenario Outlines

AKA parametrized tests. The <placeholders> in step definition as well as in the outline name, or documentation text will be filled from the tables under the Example keyword. Each row is generating separate testcase from the outline. NOTE: Scenario Outlines are not converted to parametrized testcases, but separate testcases will be generated instead. There can be multiple Example section and they can have different names, which will be reflected in the report.

Scenario Outline: add two number (<a> + <b>)

    Check if sum can add two number

    Given we have two number: <a> and <b>
    When we sum the numbers
    Then the result is: <expected>

    Examples: when both positive
        | a   | b   | expected |
        | 1   | 1   | 2        |
        | 2   | 2   | 4        |
        | 123 | 321 | 444      |

    Examples: when one negative
        | a    | b   | expected |
        | 1    | -1  | 0        |
        | 2    | -2  | 0        |
        | -123 | 321 | 198      |

Downloadable example is here

Background

The Feature can have a Background section, which will be executed before each Scenario

Background:
    Given: we have an empty DB

Scenario: add a name to the DB
    When we add new name to the DB
    Then the db has exactly one name in it

Scenario: add another name to the DB
    When we add new name to the DB
    Then the db has exactly one name in it

both expectation should pass as the Background creating an empty DB for the test

Downloadable example is here

Scenario Description

Scenario and Example descriptions are extracted and inserted into the report

Scenario Outline: add two number (<a> + <b>)

        Document your scenario here it worth it

        Given we have the avove doc
        When the test run
        Then the report contains the doc

Step arguments

Steps can get arguments either in python docstring format or in Gherkin data table format

Scenario: Arguments

Given the DB has the following data:
    | name | phone number |
    | John | +361234564   |
    | Jane | +361234561   |
    | Max  | unknown      |
When I initiate call to Max
Then I got the following error:
"""
   <H1>Error</H1>
   The phone number for Max is unknown
"""

the table or the string is passed to the function executed by the step

Downloadable example is here

Labels

Features, Scenarios, Scenario Outlines and Examples can get labels, these are available in the Testplan execution as tags, and can be used for testcase selection in a run.

@business
Feature: Important feature

    @smoke
    Scenario: base flow

        Given a business process
        When it is executed
        Then the result is expected

    @parametrized
    Scenario Outline: string reverse does not change length

        Given we have a string "<input>"
        When we reverse it
        Then the result has the same <expected>

        @short @table
        Example: short string
        | input | expected |
        | al    | 2        |
        | als   | 3        |

        @slong @table
        Example: long string
        | input      | expected |
        | 1234567890 | 10       |
        | 1234567890 | 20       |

Downloadable example is here

Non standard Gherkin Features

Special Scenarios (setup/teardown)

There are 4 scenario names, which are treated specially. They do not considered as testcases though the same bdd technics can be used to describe them. They are captured by testplan and are run at the right time to set up initial state and clean up after the test case/suite.

• setup: executed before anything in the feature file
• pre_testcase: executed before every testcase in the feature file
• post_testcase: executed after every testcase in the feature file
• teardown: executed after the whole feature file executed

@setup_teardown
Feature: setup teardown example

    Scenario Outline: Testcase <name>

        When we log Hello from Testcase <name>
        Then we log Again Hello from Testcase <name>

        Examples:
            | name  |
            | 1     |
            | 2     |
            | three |

    Scenario: setup
        And we log In setup

    Scenario: teardown
        And we log In teardown

    Scenario: pre_testcase
        And we log In pre_testcase

    Scenario: post_testcase
        And we log In post_testcase

Downloadable example is here

KNOWN_TO_FAIL

This feature is a direct hook to the Testplan feature: Xfail

One can mark a Feature or Scenario with the @KNOWN_TO_FAIL label when not possible to fix the code immediately and make the test pass. This make the test pass till it realy fails and make it failed when it start to pass to signify that the label can be removed from it. The handling of this tag is rather special it can carry some reasoning, why is it ok the test to fail. The format is @KNOWN_TO_FAIL: reason.

An example:

Feature: sum adding two number The wrong way

    @KNOWN_TO_FAIL: A simple Fail with Jira
    Scenario: add 1 and -2

        Check if 1-2 == 1

        The jira may change to be closed the find a new one please

        Given we have two number: 1 and -2
        When we sum the numbers
        Then the result is: 1

Downloadable example is here

Parallel execution

Testplan is capable of run testcases parallel within a suite, this feature is available through BDD as well. The Scenarios, Scenario Outlines or even Examples can be marked with the special @TP_EXECUTION_GROUP label together with an execution group name. Scenarios having the same execution_group will be running parallel

In the following example the positive cases will be running parallel:

Feature: Parallel execution example

    Scenario Outline: add two number

        Check if sum can add two number

        Given we have two number: <a> and <b>
        When we sum the numbers
        Then the result is: <expected>

        @TP_EXECUTION_GROUP: group1
        Examples: when both positive
            | a   | b   | expected |
            | 1   | 1   | 2        |
            | 2   | 2   | 4        |
            | 123 | 321 | 444      |

        Examples: when one negative
            | a    | b   | expected |
            | 1    | -1  | 0        |
            | 2    | -2  | 0        |
            | -123 | 321 | 198      |

More detailed docs on Testcase Parallel Execution Downloadable example is here

{{mustache}} context resolution

You can pass contextual data between your sentences in the context paramter pased to each step definition. Some times it is beneficial to even refer values from the context in your actual step definition. To do this you need to add a ContextResolver instance to your BDDTestSuiteFactory:

demo_factory = BDDTestSuiteFactory(os.path.join(base_path, 'demo'), ContextResolver())

Then you can access the content of the conext.field_name with {{field_name}} in your feature file.

@resolve @smoke
Feature: Hello world with random names

    Simple hello world example where the name is resolved from  context

    Scenario: double name

        Given a random name as firstName
        Given a random name as middleName
        When salute is called with "{{firstName}} {{middleName}}"
        Then the result is "Hello {{firstName}} {{middleName}}"

Two execution of the above can be seen below, please note that we generate the names in random, but still visible in the step definition:

  [Given a random name as firstName] -> Passed
  [Given a random name as middleName] -> Passed
  [When salute is called with "Marvin John"] -> Passed
  [Then the result is "Hello Marvin John"] -> Passed
[double name] -> Passed

  [Given a random name as firstName] -> Passed
  [Given a random name as middleName] -> Passed
  [When salute is called with "Jane Marvin"] -> Passed
  [Then the result is "Hello Jane Marvin"] -> Passed
[double name] -> Passed

The context resolver can resolve values from nested indexable structures (maps,lists) with dot notation {{person.name.lastName}} where person is a dictionary in the context, which has a name key in it, which is again a dictionary having lastName key in it. This also works with lists: myList.2.name refer to the name item of the 3rd dictionary in myList. This works out of the box for any indexable object, with the following Indexable mixin any user defined class can make indexable as the below example shows:

class Indexable:
    def __getitem__(self, item):
        return self.__dict__[item]

class indexexample(Indexable):

    def __init__(self):
        self.a = 12
        self.b = 13

context.i = indexexample()

in the fature files {{i.a}} == 12 and {{i.b}} == 13

Downloadable example is here

Step Definitions

There are two ways to provide step definitions, by default step definitions should be in the feature directory under steps subdir. All .py files are loaded by the factory. In this form all scenarios can refer all step definitions within all the .py files. This also mean if two step definition match for the same sentence, only the one will be running, which one, do not make any assumption on it.

Alternative way is to use feature files linked steps. To use this mode the factory should be created with feature_linked_steps=True In this mode each *.feature file need to have a *.steps.py pair next to it. Each feature file has it’s own set of steps so one can do different thing for the same sentence in different features. All downloadable BDD examples, except QuickStart is written in this way.

To have common steps in either mode it is possible to import step definition to other step definitions, see: Import step definitions

Step definitions are just functions which are decorated with the @step decorator (for convenience @Given, @When, @Then, @And, @But also provided). The parameter of the decorator is either a string which describe the match or a testplan.testing.bdd.parsers.Parser which can match strings.

The step functions should take at least 3 parameters:

• env: The same environment as in Multitest testcase. You have access to the drivers, for an example check: BDD rewrite of the simple tcp example
• result: A Result object same way as a Multitest testcase
• context: which is a dictionary like object. There is a separate contex for each Scenarion at runtime. Step definitions can use the context object to store/pass state between each other.

@step('say Hello World')
def step_definition(env, result, context):
    result.log('Hello World')

Step Arguments

Step arguments can be captured in the step definition as well, in this case a 4th argument will be passed to the step definition

Scenario:
    When say:
    """
    Hello World
    """

@step('say:')
def step_definition(env, result, context, arg):
    result.log(arg)

Downloadable example is here

Parameter capture

Bits and pieces from the sentence can be captured as parameter to the function

Scenario:
    When salute to John

With SimpleParser one can just enclose a name to {} and the captured string get passed to the step definition in kwargs with the same name. Deataild description of parsers see: Parsers

@step(SimpleParser('salute to {name}')
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

Step argument and Parameter capture can be combined

Scenario:
    When salute to John with:
    """
    Hello
    """

Downloadable example is in parsers example

@step(SimpleParser('salute to {name} with')
def step_definition(env, result, context, arg, name):
    result.log(arg + " " + name)

Parsers

By default the gherkin sentences matched by reg exp to step definitions, which is a powerful way of matching and capturing parameters. Some time it means more complex syntax so there is a much simpler parser to choose. The bellow two step definition doing the same thing:

Scenario:
    When salute to John

@step(SimpleParser('salute to {name}'))
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

@step(RegExParser('salute to (?P<name>.*)'))
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

Default Parser can be set on the Factory constructor

@test_plan(name="Gherkin Tests")
def main(plan):
    factory = BDDTestSuiteFactory('features', default_parser=SimpleParser)
    plan.add(MultiTest("GherkinTest", "Example Gherkin Suite", factory.create_suites()))

or within a step definition file with the follwoing way:

from testplan.testing.bdd.step_registry import step, set_actual_parser

# use the simple parser for these definitions
set_actual_parser(SimpleParser)

@step('salute to {name}')
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

@step(RegExParser('salute to (?P<name>.*)'))
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

As seen above the default always can be overwritten if passing a parser instead of a string to the decorator. Also if actual_parser is set from that point till it is set again that is the actual parser in the step definition file.

from testplan.testing.bdd.step_registry import step, set_actual_parser

# use the simple parser for these definitions
set_actual_parser(SimpleParser)

@step('salute to {name}')
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

# and the regex parser for these definitions
set_actual_parser(RegExParser)
@step('salute to (?P<name>.*)')
def step_definition(env, result, context, name):
    result.log('Hello ' + name)

Downloadable example is here

Import step definitions

It is possible to import step definition files into another step definition file. This is useful either in feature linked steps mode, when a feature file linked with a single step definition file, but one may have common steps for reuse. Or in the default mode, when several top level tests are defined, but some steps are used in several top level tests. The step_registry package has a method import_steps(path_to_package). The path should be relative to the step definition file, that importing the other. Importing multiple step definition is possible, but if steps matchers are the same only one of the steps will be executed. See example below:

one.feature

@feature_linked @smoke
Feature: Salute in English

    Scenario: Salute

        Given my name is John
        When one salute
        Then I hear: Hello John

one.steps.py

from testplan_bdd.step_registry import When, import_steps

import_steps('common.py')

@When('one salute')
def step_definition(env, result, context):
    context.salute = 'Hello {}'.format(context.name)

common.py

from testplan_bdd.step_registry import Given, Then


@Given('my name is {name}')
def step_definition(env, result, context, name):
    context.name = name

@Then('I hear: {salute}')
def step_definition(env, result, context, salute):
    result.equal(salute)(context.salute)

Downloadable example is here

Common step definitions

In case of a set of common steps, which are structured in multiple files and need to be shared between BDDTestSuiteFactories, there is an easier way besides importing them into each feature’s step directory, too. One can specify their location by giving their relative path(s) as a list in the common_step_dirs parameter of the BDDTestSuiteFactory instance:

For example, let’s assume that our directory structure looks like this:

> tree
.
|-- features
|   |-- feature1
|   |   |-- steps
|   |   |   `-- feature1.py
|   |   |-- feature11.feature
|   |   `-- feature12.feature
|   |-- feature2
|   |   |-- steps
|   |   |   `-- feature2.py
|   |   `-- feature2.feature
|   `-- steps
|       `-- features.py
`-- test_plan.py

Steps that are used by all features are defined in the features/steps directory. Therefore, only those steps that are used by a subset of the features need to be defined separately in their corresponding steps folder, in this example namely in features/feature1/steps and features/feature2/steps.

@test_plan(name="Example Gherkin Testplan")
def main(plan):
    factory1 = BDDTestSuiteFactory(
        "features/feature1",
        default_parser=SimpleParser,
        common_step_dirs=["features/steps"],
    )
    factory2 = BDDTestSuiteFactory(
        "features/feature2",
        default_parser=SimpleParser,
        common_step_dirs=["features/steps"],
    )
    plan.add(
        MultiTest(
            name="Example Gherkin Test",
            description="Example Gherkin Suites",
            suites=[*factory1.create_suites(), *factory2.create_suites()],
        )
    )

Check the downloadable example for more help.