testplan.testing.multitest.driver package¶
Subpackages¶
Submodules¶
testplan.testing.multitest.driver.app module¶
Generic application driver.
-
class
testplan.testing.multitest.driver.app.
App
(name: str, binary: str, pre_args: Union[List[Union[str, testplan.common.utils.context.ContextValue]], str, testplan.common.utils.context.ContextValue] = None, args: Union[List[Union[str, testplan.common.utils.context.ContextValue]], str, testplan.common.utils.context.ContextValue] = None, shell: bool = False, env: Dict[str, str] = None, binary_strategy: typing_extensions.Literal['copy', 'link', 'noop'][copy, link, noop] = 'link', logname: str = None, app_dir_name: str = None, working_dir: str = None, expected_retcode: int = None, sigint_timeout: int = 5, binary_log: bool = False, **options)[source]¶ Bases:
testplan.testing.multitest.driver.base.Driver
Binary application driver.
{emphasized_members_docs}
Parameters: - name – Driver name. Also uid.
- binary – Path to the application binary.
- pre_args – Arguments to be prepended to binary command. An argument
can be a
ContextValue
and will be expanded on runtime. - args – Arguments to be appended to binary command. An argument
can be a
ContextValue
and will be expanded on runtime. - shell – Invoke shell for command execution.
- env – Environmental variables to be made available to child process; context value (when referring to other driver) and jinja2 template (when referring to self) will be resolved.
- binary_strategy – Whether to copy / link binary to runpath.
- logname – Base name of driver logfile under app_path, in which Testplan will look for log_regexps as driver start-up condition. Default to “stdout” (to match the output stream of binary).
- app_dir_name – Application directory name.
- working_dir – Application working directory. Default: runpath
- expected_retcode – the expected return code of the subprocess. Default value is None meaning it won’t be checked. Set it to 0 to ennsure the driver is always gracefully shut down.
- sigint_timeout – number of seconds to wait between
SIGINT
andSIGKILL
- binary_log – if True the log_matcher will handle the logfile as binary, and need to use binary regexps. Default value is False
Also inherits all
Driver
options.-
STATUS
¶
-
abort
()¶ Default abort policy. First abort all dependencies and then itself.
-
abort_dependencies
()¶ Returns an empty generator.
-
aborted
¶ Returns if entity was aborted.
-
active
¶ Entity not aborting/aborted.
-
app_path
¶ Application directory path.
-
async_start
¶ Overrides the default async_start value in config.
-
auto_start
¶ If False, the resource will not be automatically started by its parent (generally, a Environment object) while the parent is starting.
-
binary
¶ The actual binary to execute, might be copied/linked to runpath
-
binpath
¶ ‘bin’ directory under runpath.
-
cfg
¶ Configuration object.
-
cmd
¶ Command that starts the application.
-
context
¶ Key/value pair information of a Resource.
-
context_input
(exclude: list = None) → Dict[str, Any]¶ All attr of self in a dict for context resolution
-
define_runpath
()¶ Define runpath directory based on parent object and configuration.
-
env
¶ Environment variables.
-
errpath
¶ Path for stderr file regex matching.
-
etcpath
¶ ‘etc’ directory under runpath.
-
extract_driver_metadata
() → testplan.testing.multitest.driver.base.DriverMetadata¶ Extracts driver metadata as described in the extractor function.
Returns: driver metadata
-
extract_values
() → Union[bool, testplan.common.entity.base.FailedAction]¶ Extract matching values from input regex configuration options.
-
failover
() → None¶ API to create the failover resource, to be implemented in derived class
-
fetch_error_log
() → List[str]¶ Fetch error message from the log files of driver, at first we can try self.errpath, if it does not exist, try self.logpath. Typically, several lines from the tail of file will be selected.
Returns: text from log file
-
classmethod
filter_locals
(local_vars)¶ Filter out init params of None value, they will take default value defined in its ConfigOption object; also filter out special vars that are not init params from local_vars.
Parameters: local_vars –
-
force_started
()¶ Change the status to STARTED (e.g. exception raised).
-
force_stopped
()¶ Change the status to STOPPED (e.g. exception raised).
-
hostname
¶ Returns: hostname where the ETSApp is running
-
install_files
() → None¶ Installs the files specified in the install_files parameter at the install target.
-
is_alive
¶ Called to periodically poll the resource health. Default implementation assumes the resource is always healthy.
-
log_matcher
¶ Create if not exist and return the LogMatcher object that reads the log / stdout of the driver.
Returns: LogMatcher instance
-
logger
¶ logger object
-
logname
¶ Configured logname.
-
logpath
¶ Path for log regex matching.
-
make_runpath_dirs
()¶ Creates runpath related directories.
-
name
¶ Driver name.
-
outpath
¶ Path for stdout file regex matching.
-
pause
()¶ Pauses entity execution.
-
pausing
()¶ Pause the resource.
-
pending_work
()¶ Resource has pending work.
-
pid
¶ - Return pid of the child process if available,
None
otherwise.
-
post_start
()¶ Steps to be executed right after resource is started.
-
post_stop
()¶ Steps to be executed right after resource is stopped.
-
pre_start
() → None[source]¶ Create mandatory directories and install files from given templates using the drivers context before starting the application binary.
-
pre_stop
()¶ Steps to be executed right before resource stops.
-
register_failover
(klass: testplan.common.entity.base.Entity, params: dict) → None¶ Register a failover class to instantiate if resource start fails.
Parameters: - klass – failover class
- params – parameters for failover class __init__ method
-
resolved_bin
¶ Resolved binary path from self.cfg.binary
-
restart
(clean: bool = True) → None[source]¶ Stop the driver, archive the app_dir or rename std/log, and then restart the driver.
Parameters: clean – if set to True
, perform a ‘clean’ restart where all persistence is deleted, else a normal restart.
-
resume
()¶ Resumes entity execution.
-
resuming
()¶ Resume the resource.
-
retcode
¶ Return return code of the app process or
None
.
-
runpath
¶ Path to be used for temp/output files by entity.
-
scratch
¶ Path to be used for temp files by entity.
-
start
()¶ Triggers the start logic of a Resource by executing :py:meth: Resource.starting <testplan.common.entity.base.Resource.starting> method.
-
started_check
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Predicate indicating whether a binary in a subprocess has started. Tests whether the return code is zero if the underlying binary has finished execution, otherwise tests if user-specified pattern exists in driver logs.
-
started_check_interval
¶ Driver started check interval. In practice this value is lower-bounded by 0.1 seconds.
-
status
¶ Status object.
-
stop
()¶ Triggers the stop logic of a Resource by executing :py:meth: Resource.stopping <testplan.common.entity.base.Resource.stopping> method.
-
stopped_check
() → Union[bool, testplan.common.entity.base.FailedAction]¶ Predicate indicating whether driver has fully stopped.
Default implementation immediately returns True.
-
stopped_check_interval
¶ Driver stopped check interval.
-
uid
() → str¶ Driver uid.
-
wait
(target_status, timeout=None)¶ Wait until objects status becomes target status.
Parameters: - target_status (
str
) – expected status - timeout (
int
orNoneType
) – timeout in seconds
- target_status (
-
class
testplan.testing.multitest.driver.app.
AppConfig
(**options)[source]¶ Bases:
testplan.testing.multitest.driver.base.DriverConfig
Configuration object for
App
resource.-
classmethod
build_schema
()¶ Build a validation schema using the config options defined in this class and its parent classes.
-
static
default_metadata_extractor
(driver) → testplan.testing.multitest.driver.base.DriverMetadata[source]¶
-
denormalize
()¶ Create new config object that inherits all explicit attributes from its parents as well.
-
get_local
(name, default=None)¶ Returns a local config setting (not from container)
-
ignore_extra_keys
= False¶
-
parent
¶ Returns the parent configuration.
-
set_local
(name, value)¶ set without any check
-
classmethod
testplan.testing.multitest.driver.base module¶
Driver base class module.
-
class
testplan.testing.multitest.driver.base.
Connection
(name: str, protocol: str, identifier: Union[int, str, testplan.common.utils.context.ContextValue], direction: testplan.testing.multitest.driver.base.Direction)[source]¶ Bases:
object
Base class for connection information objects.
- Such objects ideally hold data with respect to the participants in the
- connection, the ports and hosts, or the protocol.
-
class
testplan.testing.multitest.driver.base.
Direction
[source]¶ Bases:
enum.Enum
An enumeration.
-
connecting
= 'connecting'¶
-
listening
= 'listening'¶
-
-
class
testplan.testing.multitest.driver.base.
Driver
(name: str, install_files: List[Union[str, Tuple]] = None, timeout: int = 300, log_regexps: List[Pattern[AnyStr]] = None, stdout_regexps: List[Pattern[AnyStr]] = None, stderr_regexps: List[Pattern[AnyStr]] = None, file_logger: str = None, async_start: Union[testplan.common.config.base.UNSET_T, bool] = <testplan.common.config.base.UNSET_T object>, report_errors_from_logs: bool = False, error_logs_max_lines: int = 10, pre_start: Callable = None, post_start: Callable = None, pre_stop: Callable = None, post_stop: Callable = None, metadata_extractor: Callable = None, **options)[source]¶ Bases:
testplan.common.entity.base.Resource
Driver base class providing common functionality.
{emphasized_members_docs}
Parameters: - name – driver name also used as UID
- install_files – list of files to be installed
- timeout – status check timeout in seconds
- log_regexps – regexps to be matched in logfile
- stdout_regexps – regexps to be matched in stdout file
- stderr_regexps – regexps to be matched in stderr file
- file_logger – filepath for driver log, defaults to top level TP log
- async_start – whether to allow async start in environment
- report_errors_from_logs – whether to log the tail of stdout/stderr/logfile logs upon start/stop exception
- error_logs_max_lines – number of lines to be logged from the tail of stdout/stderr/logfile logs if report_errors_from_logs is True
- path_cleanup – whether to remove existing runpath elements
- pre_start – callable to execute before starting the driver
- post_start – callable to execute after the driver is started
- pre_stop – callable to execute before stopping the driver
- pre_stop – callable to execute after the driver is stopped
- metadata_extractor – callable for driver metadata extraction
Also inherits all
Resource
options.-
CONFIG
¶ alias of
DriverConfig
-
async_start
¶ Overrides the default async_start value in config.
-
errpath
¶ Path for stderr file regexp matching.
-
extract_driver_metadata
() → testplan.testing.multitest.driver.base.DriverMetadata[source]¶ Extracts driver metadata as described in the extractor function.
Returns: driver metadata
-
extract_values
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Extract matching values from input regex configuration options.
-
fetch_error_log
() → List[str][source]¶ Fetch error message from the log files of driver, at first we can try self.errpath, if it does not exist, try self.logpath. Typically, several lines from the tail of file will be selected.
Returns: text from log file
-
install_files
() → None[source]¶ Installs the files specified in the install_files parameter at the install target.
-
logpath
¶ Path for log regexp matching.
-
name
¶ Driver name.
-
outpath
¶ Path for stdout file regexp matching.
-
started_check
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Predicate indicating whether driver has fully started.
Default implementation tests whether certain pattern exists in driver loggings, always returns True if no pattern is required.
-
started_check_interval
¶ Driver started check interval. In practice this value is lower-bounded by 0.1 seconds.
-
stopped_check
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Predicate indicating whether driver has fully stopped.
Default implementation immediately returns True.
-
stopped_check_interval
¶ Driver stopped check interval.
-
class
testplan.testing.multitest.driver.base.
DriverConfig
(**options)[source]¶ Bases:
testplan.common.entity.base.ResourceConfig
Configuration object for
Driver
resource.
-
class
testplan.testing.multitest.driver.base.
DriverMetadata
(name: str, driver_metadata: Dict[KT, VT], conn_info: List[testplan.testing.multitest.driver.base.Connection] = <factory>)[source]¶ Bases:
object
Base class for holding Driver metadata.
Parameters: - name –
- driver_metadata –
- conn_info – list of connection info objects
-
testplan.testing.multitest.driver.base.
format_regexp_matches
(name: str, regexps: List[Pattern[AnyStr]], unmatched: List[T]) → str[source]¶ Utility for formatting regexp match context for rendering.
Parameters: - name – name of regexp group
- regexps – list of compiled regexps
- unmatched – list of unmatched regexps
Returns: message to be used in exception raised or empty string
testplan.testing.multitest.driver.sqlite module¶
Small wrapper driver around sqlite3 library.
-
class
testplan.testing.multitest.driver.sqlite.
Sqlite3
(name: str, db_path: str, connect_at_start: bool = True, **options)[source]¶ Bases:
testplan.testing.multitest.driver.base.Driver
Basic sqlite3 driver to add to a MultiTest environment, connect to a database and perform sql queries etc.
{emphasized_members_docs}
Parameters: - db_path (
str
) – Path to the database file to connect to. In case a relative path is provided it will be appended to the runpath. - connect_at_start (
bool
) – Connect to the database when driver starts. Default: True
-
CONFIG
¶ alias of
Sqlite3Config
-
STATUS
¶
-
abort
()¶ Default abort policy. First abort all dependencies and then itself.
-
abort_dependencies
()¶ Returns an empty generator.
-
aborted
¶ Returns if entity was aborted.
-
active
¶ Entity not aborting/aborted.
-
async_start
¶ Overrides the default async_start value in config.
-
auto_start
¶ If False, the resource will not be automatically started by its parent (generally, a Environment object) while the parent is starting.
-
cfg
¶ Configuration object.
-
context
¶ Key/value pair information of a Resource.
-
context_input
(exclude: list = None) → Dict[str, Any]¶ All attr of self in a dict for context resolution
-
db_path
¶ Database file path.
-
define_runpath
()¶ Define runpath directory based on parent object and configuration.
-
errpath
¶ Path for stderr file regexp matching.
-
extract_driver_metadata
() → testplan.testing.multitest.driver.base.DriverMetadata¶ Extracts driver metadata as described in the extractor function.
Returns: driver metadata
-
extract_values
() → Union[bool, testplan.common.entity.base.FailedAction]¶ Extract matching values from input regex configuration options.
-
failover
() → None¶ API to create the failover resource, to be implemented in derived class
-
fetch_error_log
() → List[str]¶ Fetch error message from the log files of driver, at first we can try self.errpath, if it does not exist, try self.logpath. Typically, several lines from the tail of file will be selected.
Returns: text from log file
-
fetch_table
(table, columns=None)[source]¶ Fetch a table from the db. The first row will be the column names and the following rows will be the table rows. Returns a table like:
[ ['symbol', 'amount'], ['AAPL', 12], ['GOOG', 21], ['FB', 32], ['AMZN', 5], ['MSFT', 42] ]
Parameters: - table (
str
) – Table name in the db. - columns (
list
ofstr
) – Names of columns to be fetched.
Returns: The table contents.
Return type: list
oflist
of values.- table (
-
classmethod
filter_locals
(local_vars)¶ Filter out init params of None value, they will take default value defined in its ConfigOption object; also filter out special vars that are not init params from local_vars.
Parameters: local_vars –
-
force_started
()¶ Change the status to STARTED (e.g. exception raised).
-
force_stopped
()¶ Change the status to STOPPED (e.g. exception raised).
-
install_files
() → None¶ Installs the files specified in the install_files parameter at the install target.
-
is_alive
¶ Called to periodically poll the resource health. Default implementation assumes the resource is always healthy.
-
logger
¶ logger object
-
logpath
¶ Path for log regexp matching.
-
make_runpath_dirs
()¶ Creates runpath related directories.
-
name
¶ Driver name.
-
outpath
¶ Path for stdout file regexp matching.
-
pause
()¶ Pauses entity execution.
-
pausing
()¶ Pause the resource.
-
pending_work
()¶ Resource has pending work.
-
post_start
()¶ Steps to be executed right after resource is started.
-
post_stop
()¶ Steps to be executed right after resource is stopped.
-
pre_start
() → None¶ Steps to be executed right before resource starts.
-
pre_stop
()¶ Steps to be executed right before resource stops.
-
register_failover
(klass: testplan.common.entity.base.Entity, params: dict) → None¶ Register a failover class to instantiate if resource start fails.
Parameters: - klass – failover class
- params – parameters for failover class __init__ method
-
restart
()¶ Stop and start the resource.
-
resume
()¶ Resumes entity execution.
-
resuming
()¶ Resume the resource.
-
runpath
¶ Path to be used for temp/output files by entity.
-
scratch
¶ Path to be used for temp files by entity.
-
start
()¶ Triggers the start logic of a Resource by executing :py:meth: Resource.starting <testplan.common.entity.base.Resource.starting> method.
-
started_check
() → Union[bool, testplan.common.entity.base.FailedAction]¶ Predicate indicating whether driver has fully started.
Default implementation tests whether certain pattern exists in driver loggings, always returns True if no pattern is required.
-
started_check_interval
¶ Driver started check interval. In practice this value is lower-bounded by 0.1 seconds.
-
status
¶ Status object.
-
stop
()¶ Triggers the stop logic of a Resource by executing :py:meth: Resource.stopping <testplan.common.entity.base.Resource.stopping> method.
-
stopped_check
() → Union[bool, testplan.common.entity.base.FailedAction]¶ Predicate indicating whether driver has fully stopped.
Default implementation immediately returns True.
-
stopped_check_interval
¶ Driver stopped check interval.
-
uid
() → str¶ Driver uid.
-
wait
(target_status, timeout=None)¶ Wait until objects status becomes target status.
Parameters: - target_status (
str
) – expected status - timeout (
int
orNoneType
) – timeout in seconds
- target_status (
- db_path (
-
class
testplan.testing.multitest.driver.sqlite.
Sqlite3Config
(**options)[source]¶ Bases:
testplan.testing.multitest.driver.base.DriverConfig
Configuration object for
Sqlite3
resource.-
classmethod
build_schema
()¶ Build a validation schema using the config options defined in this class and its parent classes.
-
static
default_metadata_extractor
(driver) → testplan.testing.multitest.driver.base.DriverMetadata¶
-
denormalize
()¶ Create new config object that inherits all explicit attributes from its parents as well.
-
get_local
(name, default=None)¶ Returns a local config setting (not from container)
-
ignore_extra_keys
= False¶
-
parent
¶ Returns the parent configuration.
-
set_local
(name, value)¶ set without any check
-
classmethod
testplan.testing.multitest.driver.kafka module¶
Driver for Kafka server
-
class
testplan.testing.multitest.driver.kafka.
KafkaStandalone
(name: str, cfg_template: str, binary: str = '/opt/kafka/bin/kafka-server-start.sh', host: Optional[str] = None, port: int = 0, **options)[source]¶ Bases:
testplan.testing.multitest.driver.app.App
Driver for starting a Kafka instance in standalone mode.
{emphasized_members_docs}
Parameters: - cfg_template (
str
) – Kafka config file template. - binary (
str
) – kafka-server-start.sh file path. - port (
int
) – Kafka listen port. - env (
dict
) – Environmental variables to be made available to Zookeeper process.
-
CONFIG
¶ alias of
KafkaStandaloneConfig
-
cmd
¶ Command that starts the application.
-
host
¶ Host to bind to.
-
port
¶ Port to listen on.
- cfg_template (
-
class
testplan.testing.multitest.driver.kafka.
KafkaStandaloneConfig
(**options)[source]¶ Bases:
testplan.testing.multitest.driver.app.AppConfig
Configuration object for
KafkaStandalone
resource.
testplan.testing.multitest.driver.zookeeper module¶
Driver for Zookeeper server
-
class
testplan.testing.multitest.driver.zookeeper.
ZookeeperStandalone
(name: str, cfg_template: str, binary: str = '/usr/share/zookeeper/bin/zkServer.sh', host: Optional[str] = None, port: int = 2181, env: Optional[dict] = None, **options)[source]¶ Bases:
testplan.testing.multitest.driver.base.Driver
Driver for starting a Zookeeper instance in standalone mode.
{emphasized_members_docs}
Parameters: - template (
str
) – Zookeeper config file template. - binary (
str
) – zkServer.sh file path. - port (
int
) – Zookeeper listen port. Zookeeper doesn’t support random port - env (
dict
) – Environmental variables to be made available to Zookeeper process.
-
CONFIG
¶ alias of
ZookeeperStandaloneConfig
-
connection_str
¶ Connection string.
-
env
¶ Environment variables.
-
host
¶ Host to bind to.
-
port
¶ Port to listen on.
- template (
-
class
testplan.testing.multitest.driver.zookeeper.
ZookeeperStandaloneConfig
(**options)[source]¶ Bases:
testplan.testing.multitest.driver.base.DriverConfig
Configuration object for
ZookeeperStandalone
resource.
Module contents¶
Drivers modules.
This is actually a Testplan level feature, and this subpackage could be moved to testplan.common.driver in the future.
-
class
testplan.testing.multitest.driver.
Driver
(name: str, install_files: List[Union[str, Tuple]] = None, timeout: int = 300, log_regexps: List[Pattern[AnyStr]] = None, stdout_regexps: List[Pattern[AnyStr]] = None, stderr_regexps: List[Pattern[AnyStr]] = None, file_logger: str = None, async_start: Union[testplan.common.config.base.UNSET_T, bool] = <testplan.common.config.base.UNSET_T object>, report_errors_from_logs: bool = False, error_logs_max_lines: int = 10, pre_start: Callable = None, post_start: Callable = None, pre_stop: Callable = None, post_stop: Callable = None, metadata_extractor: Callable = None, **options)[source]¶ Bases:
testplan.common.entity.base.Resource
Driver base class providing common functionality.
{emphasized_members_docs}
Parameters: - name – driver name also used as UID
- install_files – list of files to be installed
- timeout – status check timeout in seconds
- log_regexps – regexps to be matched in logfile
- stdout_regexps – regexps to be matched in stdout file
- stderr_regexps – regexps to be matched in stderr file
- file_logger – filepath for driver log, defaults to top level TP log
- async_start – whether to allow async start in environment
- report_errors_from_logs – whether to log the tail of stdout/stderr/logfile logs upon start/stop exception
- error_logs_max_lines – number of lines to be logged from the tail of stdout/stderr/logfile logs if report_errors_from_logs is True
- path_cleanup – whether to remove existing runpath elements
- pre_start – callable to execute before starting the driver
- post_start – callable to execute after the driver is started
- pre_stop – callable to execute before stopping the driver
- pre_stop – callable to execute after the driver is stopped
- metadata_extractor – callable for driver metadata extraction
Also inherits all
Resource
options.-
CONFIG
¶ alias of
DriverConfig
-
async_start
¶ Overrides the default async_start value in config.
-
errpath
¶ Path for stderr file regexp matching.
-
extract_driver_metadata
() → testplan.testing.multitest.driver.base.DriverMetadata[source]¶ Extracts driver metadata as described in the extractor function.
Returns: driver metadata
-
extract_values
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Extract matching values from input regex configuration options.
-
fetch_error_log
() → List[str][source]¶ Fetch error message from the log files of driver, at first we can try self.errpath, if it does not exist, try self.logpath. Typically, several lines from the tail of file will be selected.
Returns: text from log file
-
install_files
() → None[source]¶ Installs the files specified in the install_files parameter at the install target.
-
logpath
¶ Path for log regexp matching.
-
name
¶ Driver name.
-
outpath
¶ Path for stdout file regexp matching.
-
started_check
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Predicate indicating whether driver has fully started.
Default implementation tests whether certain pattern exists in driver loggings, always returns True if no pattern is required.
-
started_check_interval
¶ Driver started check interval. In practice this value is lower-bounded by 0.1 seconds.
-
stopped_check
() → Union[bool, testplan.common.entity.base.FailedAction][source]¶ Predicate indicating whether driver has fully stopped.
Default implementation immediately returns True.
-
stopped_check_interval
¶ Driver stopped check interval.