Scenario Editor
August 2011
support@pureload.com
Scenario Editor |
|
| PureLoad
4.3 August 2011 |
http://www.pureload.com support@pureload.com |
This chapter describes how scenarios and tasks are defined and also
the various features related to scenario management in the scenario
editor.
Scenarios of tasks is the common structure used throughout
all applications in the Pure Performance suite. A task is the actual
function that will be performed, i.e get a web page, insert data into a
database, etc. A series of tasks are defined in a scenario. An
extensive set of features is available to make tasks modular and
dynamic.
A scenario in the Pure Performance suite consists of task sequences
and tasks. The purpose with the scenario is to assemble all tasks as
they are going to be executed in any of the run time environments. I.e
PureLoad, PureAgent or PureTest.
Task sequence is basically a container that groups other task sequences and tasks. An example is a web application where a scenario shall include requests to many web pages with their specific HTML pages and graphics. All requests for a complete HTML page is in this example grouped in a task sequence:
Here we have a Scenario (Test Login) that consists of tree task
sequences, one per web-page (Start Page, Login and Logout). Each of the
task sequences consist of tasks that represent one HTTP request.
Once the PureLoad Console or PureTest application has been started a
window similar to the following appear. It consists of one main tab, Scenarios which have tree sub tabs
at the bottom of the screen:
The left pane shows the managed scenarios in tree structure. The tree
can maintain several scenarios concurrently. The menu items and tool
bar buttons are used to manage the content of the scenario tree.
Selecting an object in the tree displays the characteristics of the
object in the properties panel to the right.
The scenario tree presents objects by the value of the Name
parameter. The number within parentheses indicates the number of
iterations that will be executed. The Sequences/Tasks information below
the tree indicates the following:
Note: There is basically no difference between a scenario and a task sequence object. Scenarios can however only be defined on the top level and may also define a distribution. Scenarios are also the object type that is executed in the runtime environment.
The Name parameter that can be specified in the property panel for
scenario, task sequence and task objects need some extra attention.
Name is primarily used in the scenario editor and in the result
presentation. Its default value is the same as the selected task class
but can be specified to anything meaningful. Name is also used when
PureLoad visualizes the tasks in the the result presentation. All tasks
with the same name per scenario or task sequence will be presented as
one task in the result presentation.
An example is a scenario for load testing of a web site. The
scenario to the left will give a result tree as shown to the right:
If we are not interested in individual results for the image URLs,
we can name these tasks "image" and the results will be displayed as:
Create a new scenario by selecting the Root node in tree and select
Edit->Create Scenario or use the 
button in the tool bar:
Selecting the newly created scenario in the scenario tree shows the
properties for the scenario in the right area. The properties are:
A task sequence is a container in which tasks and other task
sequences can be grouped. There is no limit on the depth of a scenario
tree.
Select the object in the scenario tree into which the task sequence
shall be added and select the Edit->Create
Task
Sequence menu option. If the currently selected object in
the tree is a scenario or task sequence then the newly created task
sequence will be added as a child object. If a task object is currently
selected then the new task sequence will be created below the selected
task object:
Selecting the newly created task sequence in the scenario tree shows the properties for the task sequence in the right area. The properties are:
Tasks are added either to a scenario or a task sequence. Select the Edit->Create menu choice to
create a new task. If the currently selected object in the tree is a
scenario or task sequence then the newly created task will be added as
a child object. If a task object is currently selected then the new
task will be created below the selected task object.
Note: The tool bar button
creates an object in the tree
according to the current selection. If the selection is either a
scenario, task sequence or task object then the new object is always a
task.
The following dialog lists the Task Types to choose from. Select a task in the list and press Add or double click to add the task type.
The added task will now appear in the scenario editor:
All tasks have task parameters. These are grouped into generic task
parameters and task specific parameters. All tasks have the following
generic parameters:
All tasks are using the task parameter mechanism. Task parameters
defines the input interface for the task and the actual values are
specified in the scenario editor. Task parameters might be of different
types and are displayed different depending on type.
The value for all task parameters can automatically be defined using
task parameters generators and/or variables.
Scenarios of tasks are normally executed in any of the Pure
Performance suite applications. This can be PureLoad for load testing,
PureAgent for performance monitoring or the PureTest application
itself. This section describes how to execute tasks from within the
Scenario Debugger in the PureTest application.
The main purpose with the Scenario Debugger is to quickly test and
verify each individual task in a scenario. This is done by single
stepping through each task in a scenario. Detailed execution results
and execution log is shown for each task.
The debugger mini tool bar is displayed by selecting the View->Show Debug Toolbar menu
choice (if not already displayed) and its control buttons will appear
in the Scenario Editor sub tab:
The following operations are available in the debugger tool bar:
Toggle
debug/edit mode
Start debug session
) in front of the current
task
in the tree.
Stop execution
Continue/Start execution
Single step
Step Over (skip)
Toggle breakpointMake sure debug mode is enabled (use the
button in the debug tool bar).
Now press the button to start the debug session.
This operation moves the execution pointer to the first task in the
scenario which is indicated by the task pointer icon (). This pointer indicates
that the task is going to be executed during the next Single Step (), Step Over () or Continue () operation.
The task execution result is presented to right of the scenario tree:
Now you can continue to step trough the scenario analyzing the results from each task executed.
Scenarios are often designed so that they can be parameterized and
repeated many times with different data. The parameter generator
mechanism makes this possible. Each task parameter value can refer to
parameter generators to have the actual value generated during the
test. The following parameter generator types are available:
Parameter generators have five operations that they can perform:
Before assigning a parameter generator to a task parameter it needs
to be created and configured. This is done in the Parameter Generators
sub tab.
Selecting View->Create or the button in the tool bar shows a dialog with
the available data sources.
Double click on selected data source or press Add to create it. All
parameter generators have a Name parameter that must be unique. It is
recommended to set a name that match the purpose with the parameter
generator. For example User ids, Cities, Models, etc.
The counter generates numbers within specified boundaries.
Start and Stop defines the range. The special constant INF is used
for infinity (i.e the maximun number that can be represented). The
Format parameter may be used to format the generated number.
Typical examples are:
The file data source read values from a file. The file can either
contain one value per row or several values per row. Several values per
row must be delimited with a character that is used to split the values
into Elements. The file below contains 6 rows, each with 3 elements
separated with a "%" character.
| A road to heaven%Clark Moore%287 Somewhere in the sky%Vanessa Hearst%99 Man in the moon%Dirk Dustin%129 Java Server Pages%Hans Bergsten%199 Gagarin: a myth%Roland Petersson%19 Sun, Moon, Earth%Laura Craft%78 |
The following figure shows the properties for the file data source.
The books.txt file has been
loaded and the Preview Data button has be pressed. Since the specified Field Separator matches the
separator in the file will three columns appear in the grid each
representing the values in the file. The name of each element can be
changed by clicking on the table header and it is those names that are
used to access the parameter generator elements in a task parameter.
The following syntax is used in a task parameter to access a element in
a file data source:
| $Books.Prices.getNext $Books.Authors.getNext |
If having several elements defined for a file parameter generator
then each of these are managed separated from another. I.e doing a $Books.Prices.getNext will only
forward the row pointer for that specific element and not for any other.
The list parameter generator is handy when a small list of values shall be read. The management of the list is done in the List parameter generator options pane.
The calendar parameter generator is useful when generating timestamps values. The generator can be configured to increase parts of the timestamp between each getNext operation.
Having a Date Format set to yyyy-MM-dd HH:mm and Start Time set to 2007-10-12 13:37 and Increment set to 1 Hours will cause each use of the getNext operation to return the following values:
| 2007-10-12 13:37 2007-10-12 13:38 2007-10-12 13:39 etc. |
The Date Format lists a few standard formats but it can be manually edited to accommodate custom patterns.
The Unique parameter generator can be used to generate a random unique identifier in standard GUID/UUID format. Using getNext will give you (for example) the following values:
| 3cdeab54-2b6a-4fe1-bfac-b1fbe0284266 e5a01b89-61f8-45a9-b1fb-f4c8f70d38a2 53e2e70c-2c65-4b52-97af-3bab4472e2d6 etc. |
Using get operation will retreive the last used value; all other operations generates a new unique identifier.
All generators can be tested using the Test button at the bottom of the pane. The following dialog is displayed for all but File generators.
The main difference between the test dialog for file data sources is that an element (column) may be selected.
A powerful feature when testing a dynamic application is to extract information from a task, save his information to a scenario variable and use defined variables in subsequent tasks.
Variables can be used in any task parameter. The syntax of a
variable is:
| ${variable-name} |
where the value, if any, of variable-name is substituted before the
task is executed. A task parameter value can mix static text with
variables and also contain more than one variable. Assume that we have
two variables defined itemno and itemcount, then we could for example
have a URL parameter to a HTTP tasks as follows:
| http://some.where.com/getItem?item=${itemno}&count=${itemcount} |
Variables values are set by Tasks. To assign a value to a variable a task must be used. Normally this is done by extracting part of data received from a previous task, using any of the extract tasks available.
The most common use of variables is when testing a dynamic web application. Please read more about testing web applications in the Testing Web Applications guide.
But note that using the generic extract task it is possible to extract data and assign a value to a variable from any task that returns data. This includes Telnet, FTP, Mail and JDBC tasks.
Task sequences can have a condition which determines if the sequence
should be executed or not depending on previous errors or an execute
condition expression.
A scenario is normally executed task by task from beginning to end. The default behavior is to ignore errors in individual tasks and continue execution. By using the RuntimeSettingsTask the runtime settings flag "Stop On Error" can be set to false, to force the whole scenario execution to end in case of a task error.
However, sometimes it is important to execute certain tasks to perform cleanup even if an error has occurred. For this purpose, the "Stop On Error" can be overridden for a task sequence by setting the task sequence execute condition to "Always". All tasks and sub-sequences under a task sequence with condition "Always" will be executed regardless of previous errors.
Given the following scenario:
Now imagine that the CreateEntryPage results in an error. With "Stop
On Error" set to false, the execution will continue after the error and
cause the UpdatePage and the StoreEntryPage to fail as well, since no
entry was created.
If we want only the initial error to be reported as an error this can
be accomplished by setting "Stop On Error" to false. The scenario
execution will then halt right after the task 'create' has been
executed. In this case the user never logged out and the server will be
left with a user session that must time out by itself.
In order to both report only the initial error and perform logout, the
"Stop On Error" flag should be set to true and the LogoutPage task
sequence should be set to "Always". This will halt the execution after
the task 'create', but will be forced to resume for the LogoutPage task
sequence.
Task sequences can be executed conditionally by setting an execute
condition. An execute condition can be set to one of the following:
Expressions are written using standard Java language syntax with the
addition of PureLoad variables. They must evaluate to the primitive
value true or false.
PureLoad variables are automatically typed. Variables holding numerical
values will be typed as primitive double values. Variables holding
non-numerical values will be typed as strings (java.lang.String).
For example, let us say that the PureLoad variable ${foo} has a value
of "minq" and that ${bar} has a value of 17. Now the following
expressions will be evaluated as:
| Expression |
Value |
|---|---|
| ${bar} == 17 |
true |
| ${foo}.equals(“minq”) |
true |
| ${foo}.length() == ${bar} |
false |
| ${bar} > 0 && ${bar}
< 42 |
true |
| ${foo}.startsWith("m") |
true |
| ${foo}.indexOf("q") |
Error (not a boolean result) |
The Replace Task Parameters dialog is a powerful way to replace task
parameter
values for individual tasks or group of tasks. The replace
functionality can either search
for matching task parameter values or completely replace values.
Select the desired nodes in the scenario tree and select Edit->Replace Task
Parameters. All tasks that are included in the selected nodes
will
be subject for replacement in the replace task parameter dialog:
One or many task parameters (of the same type) can be selected for
replacement. If "Text to find" is specified, only task parameters
matching the specified text will be replaced. Select the "Replace All"
if all values should be replaced regardless of what they are currently
set to.
Example:
By selecting both URL task
parameters, the replace will operate on all URLs in the scenario:
Let us say that there are two tasks (one HttpGetTask and one
HttpPostTask) with URL set to http://www.somewhere.com.
The above replacement would replace the 'some' with a 'no' so that the
resulting URL becomes http://www.nowhere.com.
The Replace Scenario Parameters dialog can be used to change
distribution parameters in many scenarios. It is also possible to make
a percentage increase or decrease to distribution parameters.
Select the desired nodes in the scenario tree and select Edit->Replace Scenario
Parameters. All tasks that are included in the selected nodes
will
be subject for replacement in the replace task parameter dialog:
Example:
Making a 10%
increase in the scenarios / s for all selected scenarios with
distribution Ramp Up can be done by selecting both y0 and y1:
Note: Scenario Distribution and the Function parameter for scenario objects is only applicable when using PureLoad.
Scenario distribution is controlled by a distribution function. A
distribution function defines how many times per second a scenario
should be
executed and when the execution will start and stop. There are a number
of different functions to choose from so that the total load test
traffic can best match the desired traffic model. It is also possible
to execute without any distribution function and instead execute as
fast as possible. The available functions are described in detail below.
The combined distribution functions of all scenarios in a load test
make up the total distribution function. In order to execute the load
test exactly as specified by the total distribution function, both the
system under test and PureLoad itself must have enough resources and be
able to perform. Large systems tend to have a certain inertia and will
not respond to changes immediately. Load tests should usually be
performed over a long period of time (at least 15 minutes up to many
hours) for the distribution to take effect.
If the system under test fails to perform fast enough to handle the
traffic, the load test results will start to differ from the
distribution function. In this situation, PureLoad can act in one of
two ways: either increase the number of scenarios per second to try to
force the exact number of scenario iterations to be executed, or to
keep on following the distribution function. Which policy to use can be
set in Tool Properties -> Load
Execution -> Distribution Policy.
When the distribution policy Exact Iterations is used, PureLoad will try to compensate and push harder if the system under test performs slower than the load test distribution function. The goal is to achieve the exact number of scenario iterations as defined by the distribution function. This means that the load test will not finish until all scenario iterations have been executed.
Example:
A scenario is set up to execute 10 per second for 20 minutes (1200
seconds with a total of 12000 scenario iterations), but the tested
system can only execute 5 scenarios per second. In this case the load
test would last 40 minutes until all the 12000 scenario iterations have
been executed.
For distribution policy Follow
Distribution, PureLoad will follow the distribution regardless
of how the system under test performs. The number of executed scenario
iterations will instead be less than defined by the distribution
function. Note that Follow Distribution can not be used if any of the
scenarios in the load test do not have a function (set to function
None) and in that case Exact Iterations will be used instead.
Example:
Consider the same situation as the previous example with 10 scenario
iterations per second and a tested system which can only handle 5
scenarios per second. This time, the result would be that the load test
finishes after 20 minutes and the number of executed scenario iteraions
would be 6000.
When function is set to None, there is no function and the specified
number of scenario iterations will be executed as fast as possible.
For scenarios without distribution (function None), it is possible to
specify a weight to control the ratio between all scenarios without
distribution and with the same logical name. A weight
of 0 (zero) indicates that the scenario will be given an average
weight. The weight is usually used in combination with iterations set
to infinite. If a scenario with a limited number of iterations
finishes, the remaining scenarios will have their weights re-calculated.
Ratio examples:
Note: If a certain ratio between scenarios is desired, it is recommended that weights are used in combination with infinite iterations and that all weights are set explicitly (avoid using 0) so that the resulting ratio can be easily understood.
The scenario instances will be executed as fast as possible starting
at Start Time.
The scenarios will be distributed in a linear way over time. For
example starting with 2 scenarios per second and ending with 5
scenarios per second. Note that it is not possible to set the total
number of scenarios executed by editing the Iterations parameter. The iterations
parameter is calculated by the system using the defined parameters. The
calculated value is showed as "view only" in the iterations field.
The example above shows:
The specified number of scenarios will be distributed quadratically
starting at Start Time.
This distribution function is used to distribute the scenarios with
a Normal function.
The parameters to set for this function are:
Ramp Up is similar to Linear, but it allows the load to start from
nothing and ramp up to a certain work level. The parameter Ramp Up
controls the duration of the ramp up from zero to y0 iterations per
second.
If you are using distributions for several scenarios it is possible
to view the summary graph of all the time distributions. This is done
by selecting the Root object in the scenarios Tab. The example below
models the load on a system where there is a base load and an event
occurs which causes a peak in the load (this could be users arriving at
the office after lunch and performing some task on a server).
