Drivers

Introduction

Drivers are easy-to-use Testplan-provided interfaces for interacting with outside applications and services during testing runtime. With the help of drivers, users can easily control related applications in the scenario of integration testing.

Drivers can be used in Testplan MultiTest as well as Testplan-provided unit test framework integrations. Below is an concrete example of how drivers can be used in Testplan MultiTest.

# Example environment of three dynamically connecting drivers.
#  --------------         -----------------         ---------------
#  |            | ------> |               | ------> |             |
#  |   Client   |         |  Application  |         |   Service   |
#  |            | <------ |               | <------ |             |
#  --------------         -----------------         ---------------

...  # Other arguments passed to MultiTest constructor
environment=[
    Service(name='service'),
    Application(name='app',
                host=context('service', '{{host}}'),
                port=context('service', '{{port}}')),
    Client(name='client',
           host=context('app', '{{host}}'),
           port=context('app', '{{port}}'))
],
...  # Other arguments passed to MultiTest constructor

Testplan provides a dynamic driver configuration system, around the two following properties:

  • Drivers can be configured with attributes of other drivers.
  • Driver configuration is not required to exist until the driver starts.

Context and Context Value

In Testplan, drivers can retrieve attributes of others from the Context. Context is basically the environment which the driver belongs to, where a group of drivers serve together for the integration test. Since certain driver configuration does not exist until that driver has successfully started, Context Value are created as late-bound values which will be resolved during driver start-up, from the context, to actual configuration values.

A context value in Python code is created from a context() call, taking a driver name and a Jinja2 expression that must be a valid attribute of the corresponding driver. A context value can also be used in the configuration file of a driver with the syntax of {{context["<driver_name>"].<attribute_name>}} in the place of configuration value. The configuration file will be processed with Jinja2 template engine during driver start-up, thus it must be a valid Jinja2 template.

In practice, context values are often used for communicating hostnames, port numbers, file paths, and other such values dynamically generated at runtime among drivers. With the usage of context values, we can easily avoid collisions between setups being shared among various testing scenarios. As you might already noticed, the Application driver in the above example is using context values to retrieve the host and port of the Service driver to connect to, so does that Client driver.

Start-up schedule

In the above example, we can easily find out that the Application driver cannot actually enter the start-up procedure until the Service driver has fully started, so the Application must come after the Service, and the Client must come after the Application. Testplan provides three ways to specify the start-up schedule of the drivers:

  • Pass a list of drivers to the environment parameter. In this case, drivers will be scheduled sequentially. The driver comes later in the passed-in list will not be started until the earlier one has fully started.
  • Pass a list of drivers to the environment parameter, while some of them has async_start parameter set to True. This case is similar to the previous one except that drivers after such an async_start driver will be scheduled to start even that driver has not fully started.
  • Pass a list of drivers to the environment parameter, and pass a dictionary to the dependencies parameter. In this case, Testplan will work out a feasible schedule meeting all the dependency (driver A must start before driver B) requirements while tend to start more drivers simultaneously, and the order of drivers in environment list will be ignored. For a key-value pair in the dependencies dictionary, the drivers on the key side should always be fully started before scheduling the drivers on the value side, i.e. left-hand side are before right-hand side.

The third way (making use of dependencies parameter) is usually recommended since the overall test runtime could be possibly reduced with drivers being started simultaneously. The above example can be changed as following using dependencies:

# Example environment of three dynamically connecting drivers.
#  --------------         -----------------         ---------------
#  |            | ------> |               | ------> |             |
#  |   Client   |         |  Application  |         |   Service   |
#  |            | <------ |               | <------ |             |
#  --------------         -----------------         ---------------

# Outside MultiTest constructor
client = Client(name='client',
                host=context('app', '{{host}}'),
                port=context('app', '{{port}}'))
application = Application(name='app',
                          host=context('service', '{{host}}'),
                          port=context('service', '{{port}}'))
service = Service(name='service')
# Outside MultiTest constructor

# Inside MultiTest constructor
...  # Other arguments passed to MultiTest constructor
environment=[
    client, application, service  # Order no longer matters
],
dependencies={
    service: application,
    application: client
},
...  # Other arguments passed to MultiTest constructor
# Inside MultiTest constructor

Another example containing simultaneous driver start-up can be found here.

Work with unit test

Drivers can also be useful while working with other unit testing frameworks like like GTest or Hobbes Test. Testplan will export environment variables for newly started test process. Have a look at the following code:

plan.add(GTest(
    name='My GTest',
    binary=BINARY_PATH,
    environment=[
        TCPServer(name='my server'),
        TCPClient(name='client-101',
            host=context('server', '{{host}}'),
            port=context('server', '{{port}}')
        )
    ]
)

In your unit test process, you can find an environment variable named DRIVER_MY_SERVER_ATTR_HOST, likewise, DRIVER_CLIENT_101_ATTR_PORT is also available. It is easy to understand that the string is formatted in uppercase, like DRIVER_<uid of driver>_ATTR_<attribute name>, while hyphens and spaces are replaced by underscores.

Built-in drivers

  • Driver baseclass which provides the most common functionality features and all other drivers inherit .
  • App that handles application binaries. See an example demonstrating how App driver can be used on an fxconverter python application here.
  • TCPServer and TCPClient to create TCP connections on the Multitest local environment and can often used to mock services. See some examples here.
  • ZMQServer and ZMQClient to create ZMQ connections on the Multitest local environment. See some examples demonstrating PAIR and PUB/SUB connections here.
  • FixServer and FixClient to enable FIX protocol communication i.e between trading applications and exchanges. TLS also supported if passing a TLSConfig. See some examples demonstrating FIX communication here.
  • HTTPServer and HTTPClient to enable HTTP communication. See some examples demonstrating HTTP communication here.
  • Sqlite3 to connect to a database and perform sql queries etc. Examples can be found here.
  • ZookeeperStandalone to start one standalone Zookeeper instance. Examples can be found here.
  • KafkaStandalone to start one standalone Kafka instance. Examples can be found here.

Custom

New drivers can be created to drive custom applications and services, manage database connections, represent mocks etc. These can inherit existing ones (or the base Driver) and customize some of its methods i.e (__init__, starting, stopping, etc). The Driver base class contains most common functionality that a MultiTest environment driver requires, including ability to provide file templates that will be instantiated using the context information on runtime and mechanisms to extract values from logfiles to retrieve dynamic values assigned (like host/port listening).

A generic Application driver inherits the base driver class and extends it with logic to start/stop a binary as a sub-process.

Here is a custom driver that inherits the built-in App driver and overwrites App.post_start method to expose host and port attributes that was written in the logfile by the application binary.

from testplan.testing.multitest.driver.app import App

class ServerApp(App):

    def __init__(self, **options):
        super(ServerApp, self).__init__(**options)
        self.host = None
        self.port = None

    def post_start(self):
        super(ServerApp, self).post_start()
        # In this example, log_regexps contain:
        #     re.compile(r'.*Listener on: (?P<listen_address>.*)')
        # and the logfile will contain a line like:
        #     Listener on: 127.0.0.1:10000
        self.host, self.port = self.extracts['listen_address'].split(':')
        # so self.host value will be: '127.0.0.1'
        # and self.port value will be: '10000'

See also the full downloadable example for this custom app.

Driver Interconnections

Connection between drivers can be visualised through the --driver-info flag in the cli. The connections are defined in the drivers’ metadata. When the --driver-info flag is enabled, Testplan will extract connections from each driver and format them into a flow chart visible on the web report. The built-in drivers already have their connections defined and you can add new connections by overriding the EXTRACTORS attribute for each Driver subclass.

Testplan defines 2 types of connections by default, PortDriverConnectionGroup and FileDriverConnectionGroup.

Drivers that inherit App will automatically search for the its network and file connections via the SubprocessPortConnectionExtractor and the SubprocessFileConnectionExtractor. These extractors use psutil functions to extract connections.

New types of driver connections can also be defined. To do so, you will need to create 3 new classes that inherits BaseConnectionInfo, BaseDriverConnectionGroup and BaseConnectionExtractor.

Here is an example to define a new type of connection.

from testplan.testing.multitest.driver.connection import Direction, BaseConnectionInfo, BaseDriverConnectionGroup

class NewConnectionInfo(BaseConnectionInfo):
    @property
    def connection_rep(self):
        return f"new://{self.identifier}" # this is the value by which drivers will be matched

    def promote_to_connection(self):
        return NewDriverConnectionGroup.from_connection_info(self)

class NewDriverConnectionGroup(BaseDriverConnectionGroup):
    @classmethod
    def from_connection_info(cls, driver_connection_info: BaseConnectionInfo):
        # Add any custom logic if needed here.
        # For example, to add a dummy driver
        conn = super(NewDriverConnection, cls).from_connection_info(driver_connection_info)
        conn.in_drivers["Dummy"].add("")
        return conn

    def add_driver_if_in_connection(self, driver_name: str, driver_connection_info: NewConnectionInfo):
        # Define any logic on how to match drivers here
        # Default behavior for predefined Connections is to match based on connection attribute
        if self.connection == driver_connection_info.connection:
            # Add the drivers here
            self.out_drivers[driver_name].append(SOME_INFO)
            return True
        return False

class NewConnectionExtractor(BaseConnectionExtractor):
    def extract_connections(self, driver):
        return [
            NewConnectionInfo(
                name="Example",
                service="Example",
                protocol="Example",
                identifier=driver.attribute,
                direction=Direction.LISTENING,
            )
        ]

To use the new connection, override EXTRACTORS in the relevant Driver class.

from testplan.testing.multitest.driver.base import Driver, DriverMetadata, Direction

class NewDriver(Driver):
    EXTRACTORS = [NewConnectionExtractor()]

See the example for more information here.