Tests

From IRON Test Suite
Jump to navigation Jump to search

Tests are complete scenarios which include a succession of R2D2 commands, macros, analyze of results, conditions of test branching and exit. Once created, a test can be integrated within the Scheduler to trig and run the tests. The Tests window allows the navigate in your test tree with the Test Browser but also to develop your own tests with the Test Editor.

TESTS BROWSER

The Tests window displays two panes: the left one with a tree of folders, the right one with the list of tests for the selected folder.

Tests window.png

Folder Pane

The root of the folders shows a tree of the Iron Suite partitions. The view of the root depends of the user access level. If the user is the Iron Suite system administrator, he has the complete view and access to all partitions by default and can navigate into the complete tree: when he selects one of the partitions, then the view and access is restricted to the tree dedicated to this given partition only.

Folder Pane.png

Under the given partition branch, the folder list can dropped-down to 3 levels, and the tests can be associated to each of these 3 levels. The name of the test must be unique into the same folder, but same name can be used into another tree branch. In order to manage the folders, 3 buttons are available into the header: Add, Edit, Delete

  • Add button to create a folder into the current branch of the tree.
  • Rename button to rename the selected folder, and/or duplicated into another branch of the tree.
  • Delete button to delete the selected folder. In such a case, all the attached tests are removed as well.

Test Pane

The right pane displays the list of existing tests within the selected folder with several configurable columns like:

  • Name (must be unique into the given folder)
  • Version
  • Description
  • Status: to display if the test is currently running in one of the scheduler tasks.
  • Command count: to display the number of steps executed.
  • Run OK: to display the number of tests run with success.
  • Run KO: to display the number of tests run with error
  • Last Error


In order to manage the tests, 5 buttons are available in the header: Add, Edit, Delete, Import, Export

  • Add button allows to create a new test with the Test Editor.
  • Edit button (or a double click) allows to open an existing test in the #Test Editor. It is disabled if the test is currently edited by another user or running within a scheduler task.
  • Delete button allows to delete a test. It is disabled if the test is under use in a scheduler task or edited by another user.
  • Import button allows to import Tests from text file format.
  • Export button allows to export selected Tests text files to export to another Test Manager.


TEST EDITOR

The Test editor allows to compose easily tests. This window is composed by test header and a test grid.

Test Editor.PNG



Test Header

The test path is displayed into the window bar name. The Test name, a Version number and a test Description can be defined in free field on the top box of the window.
Here the Data Collection is a set of variables/parameters previously defined that will be used for the entire test and all the macros and analysis actions.
On the top left, four buttons are available to manage the rows:

  • Add button allows to add a line at the end of the test.
  • Insert button is used to insert a new row above the current one.
  • Delete button is used to remove the selected row. If any, the upper and lower rows are checked to eventually remove not coherent lines
  • Disable button allows to disable selected row. If the row select is already disable, the row will be enabled. The disable row appears underlined in light blue writed in green as a comment.

When a new row is added or inserted, the ‘Action’ window opens to select the proper action. Then if the ‘Cancel’ button is clicked, the row insertion is canceled, and if the ‘Save’ button is clicked the new action is inserted as a new row. The global context is automatically controlled in order to validate the action insertion in the test with the existing upper and lower rows to guaranty the test coherence.

Test Grid


The 3 Robots and Ports drop-down lists are used to assign the corresponding iQsim Robot and its port the tasks is to be executed from. This Robot & Port selection is used by all the macros that are executed in the corresponding task (column). The Robot/Port selection must be unique into the same test.



The test grid is divided into 6 columns and an unlimited number of rows: each column figures a complete test script on dedicated port and each row a new step in the test.

Column 1

This is the step number, this reference can be used for jump or analysis.

Column 2 : Action

This is used to select the type of action to perform from the drop-down list. It means that the user decides to run macros (EXECUTE/RUN), get macro results (GET RESULT), analyze results (ANALYSIS), manipulate variables of the data collection, or decides to branch the test process. The list of actions available is listed below:


  • The third column named parameters is automatically fill when you configure your Action with the step editor.
  • The columns #A Cmd/Macro, #B Cmd/Macro and #C Cmd/Macro are used to select the command or the macro to be executed in the corresponding test: three commands and/or macros can be executed in parallel.

A double-click on the selected row in Action column will allow to change the action of this row. The double-click elsewhere will allow to change the parameters of the action.

GET RESULT Step editor EXECUTE Step editor EXIT COND FAIL Step editor
Get Result CDR.png
Execute.png
Conditional exit.png


Colors are used to enhanced the readability of scripts :
Blue : R2D2 commands or Macro
Highlighted Green : Exit pass instruction
Highlighted Red : Exit fail instruction
Italic green : Comment
Highlighted Gray italic green : Disabled line

ACTION LIST

EXECUTE

Description

EXECUTE action allows to run defined robot Macros.
For this action, three combo boxes show the list of predefined macros (One per port): up to 3 macros can be executed concurrently.

Execute.PNG

Several EXECUTE actions can be used successively.

Click on the button Pencil.png allows to edit the selected macro or create a new macro if "None" is selected.

RUN

Description

RUN action is used to execute a child test inside/from a parent test as an included procedure. There is no limitation of number of RUN action in a parent test.
It fully executes an existing test in immediate mode. The parent test execution is stopped and the execution continues with the child test.
When the child test ends, the parent test continues to its own next step.

  • If the child test is stopped by the Iron Test Manager due to an internal or a syntax error, then the parent test is stopped at the same time.


RUN action is limited to one level (checked during execution) and do not support recursion: during the process, if the Iron Test Manager finds a RUN command inside a child test, then it stops with an execution error.
Two parameters are available to set this action:

  • Two radio-buttons to select the context to run the child test in terms of Data Collection:
    • 'Parent Context' : Data Collection defined in Child test will be ignored and the Parent Data Collection will be used.
    • 'Child Context' : Data Collection defined in Child test will be used.
  • The name of the child test to be executed : a combo box gives the list of existing tests with their name, except the parent test itself.


Run.png



VOICE

Description

VOICE action allows to perform a PESQ or POLQA test on 2 voice samples. These two samples are from Play Voice macro and a Stop Record macro.

Voice.png


The result is automatically generate in a text file


Voice Analysis report



GET RESULT

Description

GET RESULT action get the results of a macro (EXECUTE action), of a test (RUN action) or of a voice analysis (VOICE action). The result can be a direct result from a Macro, Test or Voice Analysis or extracted from the CDR file generated by the iQsim robot. The return value by a GET RESULT action can be used as a CONDITION action input.


GET RESULT from CDR after EXECUTE action (Execute CDR)

GetResult CDR.PNG
Field name Description
Origin Step or Cell It is a droplist with the cell coordinates where is the macro to get the result from.
CDR Field The droplist of the field available in the CDR.
CDR Index The index of the CDR generated by the macro. If this parameter is missing the last CDR generated by the macro is considered. If several CDR were generated by the macro, this is the position of the CDR starting from 1 to n where 1 is the first (oldest) CDR and n the last (latest).
Result Variable The destination variable to store the result. This variable must be defined in the Data Collection selected for the test.


GET RESULT from Macro result after EXECUTE action (Execute)

GetResult Execute.PNG
Field name Description
Origin Step or Cell It is a droplist with the cell coordinates where is the macro to get the result from.
Result Number The index of the macro result. If this parameter is missing, the last result of the macro is considered.
Variable The destination variable to store the result. This variable must be defined in the data collection selected for the test


GET RESULT from sub Test execution after RUN action (Run)

GetResult Run.PNG
Field name Description
Origin Step or Cell It is a droplist with the cell coordinates where is the subtest to get the result from.
Variable The destination variable to store the result. This variable must be defined in the data collection selected for this test


GET RESULT from Voice Analysis after VOICE action (Voice)

Get Result Voice.png
Field name Description
Origin Step or Cell It is a droplist with the cell coordinates where is the Voice Analysis to get the result from.
Variable The destination variable to store the result. This variable must be defined in the data collection selected for this test


Example

The example below shows how to get the result of the macro running in #A8 and to compare the result with the string POWER_OFF.

After Get Result.png



CONDITION

Description

CONDITION action which will be used to analyze or compare values.

Condition.png

3 fields are configurable:

  • the type of action (COMPARE/SUM/DIFFERENCE)
Action Type Description
COMPARE This function compares Parameter 1 and Parmeter 2
SUM This fonction returns the sum of Parameter 1 and Parameter 2
DIFFERENCE This fonction returns the difference between Parameter 1 minus Parameter 2
  • 2 parameter fields. Parameter type can be a variable, a constant or a system constant.


Example

The example below shows the CONDITION action used to COMPARE a variable ($Result) with a string constant (POWER_OFF).

After Get Result.png



GOTO

Description

GOTO action is used to branch conditionally the test process to a specific step. It could be used to create a loop by branching backward or to jump actions by branching forward.
GOTO can be selected after a CONDITION action only (Except for Unconditional GOTO). The branch condition is calculated with the result of the previous step.

Goto.png

3 fields allows to configure this action:

  • The Condition selection type is the branching condition
Condition Description
Unconditional branch to the 'Coordinate' field is automatic without condition
Equal branch will occurs if the previous comparison of CONDITION action is equal.
Different branch will occurs if the previous comparison of CONDITION action is not equal.
Higher branch will occurs if the result is higher than the 'condition parameter'
Lower branch will occurs if the result is lower than the 'condition parameter'
  • The Destination step field allows to specify where the test will execute the next action, in case of branching.
  • The Condition Parameter is the parameter used to resolve the CONDITION that can be a constant (numeric or text) or a variable from the Data Collection. For the conditions EqualQUAL & Different, the Condition Parameter is by default set to @TRUE. A @FALSE or 0 value will oppose the result.


Example

The example shows how the test branches to row 12 if the output parameter of cell #C12 contained in the variable $Result is NOT EQUAL to the string POWER_OFF.

Example Goto.png



EXIT

EXIT action is used to immediatly end a test iteration. If it is placed just after a RUN action or a EXECUTE action, it can be conditional. In all other case, it is unconditional.

Unconditional EXIT

Whatever the status (PASS or FAIL), an unconditional EXIT has 2 fields;

  • A numeric exit code (Variable, constant or system constant)
  • A string exit cause (Variable or constant)
Unconditional exit.png



Conditional EXIT

If the conditional EXIT occurs just after a RUN action it will be active following the result of the subtest (PASS or FAIL)

  • Subtest Result
  • A numeric exit code (Variable, constant or system constant)
  • A string exit cause (Variable or constant)

if the conditional EXIT occurs after an EXECUTE, a combination of the status of the 3 ports will determinate if the test end or not.

  • The macro results can be selected from the 3 droplists. Exit condition can be 'FAIL', 'PASS' or 'DON’T CARE' (the macro result is ignore).
  • A numeric operator choice with AND and OR to combine the macro results
  • A numeric exit code (Variable, constant or system constant)
  • A string exit cause (Variable or constant)
Conditional exit.png



SET

SET action is used to assign value to a variable or to swap the value of 2 variables. In the case where a value is assign to a variable, the value can be numeric, text, variable from Data Collection or a system constant.

Set.png



NUM

NUM action allows to perform all standard arithmetic operations like addition, substraction, multiplication or euclidean division.

The 2 operands of the calculation can be a number, a variable from the data collection or a system constant.
The operator is select with a radio button between Add(+), Sub(-), Mul(x) or Div (:)
The result can only be a variable.

For these 3 parameters, a droplist shows variable available in the data collection.

Num.png



FLUSH

Flush action is used to flush the current value of the Data Collection maintained in memory to the Data Collection recorded value into the database.
It means that next usage of the current Data Collection will start with the current memory values of the variables.
The available parameter can be:

  • 'ALL' to flush the entire Data Collection,
  • Or a sequence of variable names separated by a comma to flush only a part of the Data Collection. Flushing variables of a parent Data Collection from a child test is not permitted.
Flush.png



TEXT

The TEXT action allows to perform 5 different string manipulations.

Merge

With this function, 2 different values or variables or system constants can be concatenate in the same variable.

Merge.png


Cut

The Input variable will be cut from value in the Start field for content of the Length field. The result will be store in the variable in the Output variable.

Cut.png


Find

This function allows to find the string specified in the Searched text field in the string defined in Input variable field. Location of 1st character will be stored in the Output variable.

Find.png


Take

This function allows to extract a specific value in string with separator. The value in the place Rank separated with Begin separator (optionally End separator in the Input value will be stored in the Output value.

Take.png


Length

Length of Input variable will be stored in Output variable.

Length.png


Action COMMENT

START & END TEST

Start Test

Every time a test is run by the Scheduler, a new line is inserted into the ‘Results’ pane with the pseudo action START TEST.
This line is always visible even if the test is run in quiet mode (see Scheduler chapter): same, if the test is running as a child of a parent test with the ‘RUN’ action when the parent test is in verbose mode.
If the test runs in quiet mode, it generates 2 lines only into the ‘Results’ pane, the START TEST and the END TEST actions.

End Test

All the tests must end with a conditional or unconditional EXIT action to get the test result determined. This condition guaranty that a test global result line is always inserted in the 'Results' pane.
The test result is the global status of the test execution and will generate a specific line into the 'Results' pane. This line is generated by the conditional EXIT action, and optionally displays the 'Exit Code' and 'Exit Cause' as well.
In the case of test error during the test execution, the test can interrupted due to a syntax error or any other case of break reasons. The scheduler process generates a pseudo EXIT FAIL result with a specific exit code and cause to differentiate this test break from the normal execution of a real EXIT action.
If a test is runs with the 'RUN' action from another test, the 'Exit Code' and 'Exit Cause' can be reused into the calling test with a GET RESULT action in TEST mode, and a result index of 1 for Code and 2 for Cause: this solution allows to copy the result of a child test into the parent’s variables and to branch the parent test conditionally even if the data collection is different between parent & child tests.