Noteworthy in Version 1.2.5¶
Scenario Outline: Better represent Example/Row¶
Since: | behave 1.2.5a1 |
---|---|
Covers: | Name annotation, file location |
A scenario outline basically a parametrized scenario template. It represents a macro/script that is executed for a data-driven set of examples (parametrized data). Therefore, a scenario outline generates several scenarios, each representing one example/row combination.
# -- file:features/xxx.feature
Feature:
Scenario Outline: Wow # line 2
Given an employee "<name>"
Examples: Araxas
| name | birthyear |
| Alice | 1985 | # line 7
| Bob | 1975 | # line 8
Examples:
| name | birthyear |
| Charly | 1995 | # line 12
Up to now, the following scenarios were generated from the scenario outline:
Scenario Outline: Wow # features/xxx.feature:2
Given an employee "Alice"
Scenario Outline: Wow # features/xxx.feature:2
Given an employee "Bob"
Scenario Outline: Wow # features/xxx.feature:2
Given an employee "Charly"
Note that all generated scenarios had the:
- same name (scenario_outline.name)
- same file location (scenario_outline.file_location)
From now on, the generated scenarios better represent the example/row combination within a scenario outline:
Scenario Outline: Wow -- @1.1 Araxas # features/xxx.feature:7
Given an employee "Alice"
Scenario Outline: Wow -- @1.2 Araxas # features/xxx.feature:8
Given an employee "Bob"
Scenario Outline: Wow -- @2.1 # features/xxx.feature:12
Given an employee "Charly"
Note that:
- scenario name is now unique for any examples/row combination
- scenario name optionally contains the examples (group) name (if one exists)
- each scenario has a unique file location, based on the row’s file location
Therefore, each generated scenario from a scenario outline can be selected via its file location (and run on its own). In addition, if one fails, it is now possible to rerun only the failing example/row combination(s).
The name annoations schema for the generated scenarios from above provides the new default name annotation schema. It can be adapted/overwritten in “behave.ini”:
# -- file:behave.ini
[behave]
scenario_outline_annotation_schema = {name} -- @{row.id} {examples.name}
# -- REVERT TO: Old naming schema:
# scenario_outline_annotation_schema = {name}
The following additional placeholders are provided within a scenario outline to support this functionality. They can be used anywhere within a scenario outline.
Placeholder | Description |
---|---|
examples.name | Refers name of the example group, may be an empty string. |
examples.index | Index of the example group (range=1..N). |
row.index | Index of the current row within an example group (range=1..R). |
row.id | Shortcut for schema: “<examples.index>.<row.index>” |
Scenario Outline: Name may contain Placeholders¶
Since: | behave 1.2.5a1 |
---|
A scenario outline can now use placeholders from example/rows in its name or its examples name. When the scenarios a generated, these placeholders will be replaced with the values of the example/row.
Up to now this behavior did only apply to steps of a scenario outline.
EXAMPLE:
# -- file:features/xxx.feature
Feature:
Scenario Outline: Wow <name>-<birthyear> # line 2
Given an employee "<name>"
Examples:
| name | birthyear |
| Alice | 1985 | # line 7
| Bob | 1975 | # line 8
Examples: Benares-<ID>
| name | birthyear | ID |
| Charly | 1995 | 42 | # line 12
This leads to the following generated scenarios, one for each examples/row combination:
Scenario Outline: Wow Alice-1985 -- @1.1 # features/xxx.feature:7
Given an employee "Alice"
Scenario Outline: Wow Bob-1975 -- @1.2 # features/xxx.feature:8
Given an employee "Bob"
Scenario Outline: Wow Charly-1885 -- @2.1 Benares-42 # features/xxx.feature:12
Given an employee "Charly"
Scenario Outline: Tags may contain Placeholders¶
Since: | behave 1.2.5a1 |
---|
Tags from a Scenario Outline are also part of the parametrized template. Therefore, you may also use placeholders in the tags of a Scenario Outline.
Note
- Placeholder names, that are used in tags, should not contain whitespace.
- Placeholder values, that are used in tags, are transformed to contain no whitespace characters.
EXAMPLE:
# -- file:features/xxx.feature
Feature:
@foo.group<examples.index>
@foo.row<row.id>
@foo.name.<name>
Scenario Outline: Wow # line 6
Given an employee "<name>"
Examples: Araxas
| name | birthyear |
| Alice | 1985 | # line 11
| Bob | 1975 | # line 12
Examples: Benares
| name | birthyear | ID |
| Charly | 1995 | 42 | # line 16
This leads to the following generated scenarios, one for each examples/row combination:
@foo.group1 @foo.row1.1 @foo.name.Alice
Scenario Outline: Wow -- @1.1 Araxas # features/xxx.feature:11
Given an employee "Alice"
@foo.group1 @foo.row1.2 @foo.name.Bob
Scenario Outline: Wow -- @1.2 Araxas # features/xxx.feature:12
Given an employee "Bob"
@foo.group2 @foo.row2.1 @foo.name.Charly
Scenario Outline: Wow -- @2.1 Benares # features/xxx.feature:16
Given an employee "Charly"
It is now possible to run only the examples group “Araxas” (examples group 1) by using the select-by-tag mechanism:
$ behave --tags=@foo.group1 -f progress3 features/xxx.feature
... # features/xxx.feature
Wow -- @1.1 Araxas .
Wow -- @1.2 Araxas .
Scenario Outline: Run examples group via select-by-name¶
Since: | behave 1.2.5a1 |
---|
The improvements on unique generated scenario names for a scenario outline (with name annotation) can now be used to run all rows of one examples group.
EXAMPLE:
# -- file:features/xxx.feature
Feature:
Scenario Outline: Wow # line 2
Given an employee "<name>"
Examples: Araxas
| name | birthyear |
| Alice | 1985 | # line 7
| Bob | 1975 | # line 8
Examples: Benares
| name | birthyear |
| Charly | 1995 | # line 12
This leads to the following generated scenarios (when the feature is executed):
Scenario Outline: Wow -- @1.1 Araxas # features/xxx.feature:7
Given an employee "Alice"
Scenario Outline: Wow -- @1.2 Araxas # features/xxx.feature:8
Given an employee "Bob"
Scenario Outline: Wow -- @2.1 Benares # features/xxx.feature:12
Given an employee "Charly"
You can now run all rows of the “Araxas” examples (group) by selecting it by name (name part or regular expression):
$ behave --name=Araxas -f progress3 features/xxx.feature
... # features/xxx.feature
Wow -- @1.1 Araxas .
Wow -- @1.2 Araxas .
$ behave --name='-- @.* Araxas' -f progress3 features/xxx.feature
... # features/xxx.feature
Wow -- @1.1 Araxas .
Wow -- @1.2 Araxas .
Exclude Feature/Scenario at Runtime¶
A test writer can now provide a runtime decision logic to exclude a feature, scenario or scenario outline from a test run within the following hooks:
before_feature()
for a featurebefore_scenario()
for a scenario- step implementation (normally only: given step)
by using the skip()
method before a feature or scenario is run.
# -- FILE: features/environment.py
# EXAMPLE 1: Exclude scenario from run-set at runtime.
import sys
def should_exclude_scenario(scenario):
# -- RUNTIME DECISION LOGIC: Will exclude
# * Scenario: Alice
# * Scenario: Alice in Wonderland
# * Scenario: Bob and Alice2
return "Alice" in scenario.name
def before_scenario(context, scenario):
if should_exclude_scenario(scenario):
scenario.skip() #< EXCLUDE FROM RUN-SET.
# -- OR WITH REASON:
# reason = "RUNTIME-EXCLUDED"
# scenario.skip(reason)
# -- FILE: features/steps/my_steps.py
# EXAMPLE 2: Skip remaining steps in step implementation.
from behave import given
@given('the assumption "{assumption}" is met')
def step_check_assumption(context, assumption):
if not is_assumption_valid(assumption):
# -- SKIP: Remaining steps in current scenario.
context.scenario.skip("OOPS: Assumption not met")
return
# -- NORMAL CASE:
...
Active Tags: Use “@only.with_{category}”¶
The term active tags is used for tags where it is decided at runtime if a tag is enabled or disabled. The runtime logic excludes then scenarios/features with disabled tags.
Assuming you have the feature file where:
- scenario “Alice” should only run when browser “Chrome” is used
- scenario “Bob” should only run when browser “Safari” is used
# -- FILE: features/alice.feature
Feature:
@only.with_browser=chrome
Scenario: Alice (Run only with Browser Chrome)
Given I do something
...
@only.with_browser=safari
Scenario: Bob (Run only with Browser Safari)
Given I do something else
...
# -- FILE: features/environment.py
# EXAMPLE: ACTIVE TAGS, exclude scenario from run-set at runtime.
# NOTE: OnlyWithCategoryTagMatcher implements the runtime decision logic.
from behave.tag_matcher import OnlyWithCategoryTagMatcher
import os
import sys
active_tag_matcher = None
def before_all(context):
# -- SETUP ACTIVE-TAG MATCHER: For category="browser"
global active_tag_matcher
current_browser = os.environ.get("BEHAVE_BROWSER", "chrome")
active_tag_matcher = OnlyWithCategoryTagMatcher("browser", current_browser)
def before_scenario(context, scenario):
# -- NOTE: scenario.effective_tags := scenario.tags + feature.tags
if active_tag_matcher.should_exclude_with(scenario.effective_tags):
# -- NOTE: Exclude any with @only.with_browser=<other_browser>
scenario.skip(reason="DISABLED ACTIVE-TAG")
Note
By using this mechanism, the @only.with_browser=*
tags become
active tags. The runtime decision logic decides when these tags
are enabled or disabled (and uses them to exclude their scenario/feature).
Active Tags: Use “@only.with_{category}” with Many Categories¶
When you use many categories
for active tags, it becomes unnecessary
complicated with the earlier described mechanism. In this case, you should use
the OnlyWithAnyCategoryTagMatcher
.
Assuming you have scenarios with the following runtime conditions:
- Run scenario Alice only on Windows OS
- Run scenario Bob only with browser Chrome
# -- FILE: features/alice.feature
# TAG SCHEMA: @only.with_{category}={current_value}
Feature:
@only.with_os=win32
Scenario: Alice (Run only on Windows)
Given I do something
...
@only.with_browser=chrome
Scenario: Bob (Run only with Web-Browser Chrome)
Given I do something else
...
# -- FILE: features/environment.py
from behave.tag_matcher import OnlyWithAnyCategoryTagMatcher
import sys
# -- MATCHES ANY TAGS: @only.with_{category}={value}
# NOTE: category_value_provider provides active values for categories.
category_value_provider = {
"browser": os.environ.get("BEHAVE_BROWSER", "chrome"),
"os": sys.platform,
}
active_tag_matcher = OnlyWithAnyCategoryTagMatcher(category_value_provider)
def before_feature(context, feature):
if active_tag_matcher.should_exclude_with(feature.tags):
feature.skip(reason="DISABLED ACTIVE-TAG")
def before_scenario(context, scenario):
if active_tag_matcher.should_exclude_with(scenario.effective_tags):
scenario.skip("DISABLED ACTIVE-TAG")
Note
Unknown categories, missing in the category_value_provider
are ignored.
Test Stages: Use different Step Implementations for Each Stage¶
A test stage allows the user to provide different step and environment implementation for each stage. Examples for test stages are:
- develop (example: development environment with simple database)
- product (example: use the real product and its database)
- systemint (system integration)
- ...
Each test stage may have a different test environment and needs to fulfill different testing constraints.
EXAMPLE DIRECTORY LAYOUT (with stage=testlab
and default stage):
features/
+-- steps/ # -- Step implementations for default stage.
| +-- foo_steps.py
+-- testlab_steps/ # -- Step implementations for stage=testlab.
| +-- foo_steps.py
+-- environment.py # -- Environment for default stage.
+-- testlab_environment.py # -- Environment for stage=testlab.
+-- *.feature
To use the stage=testlab
, you run behave with:
behave --stage=testlab ...
or define the environment variable BEHAVE_STAGE=testlab
.