APIs
     
Subsystem
list
     
Top
level

Standards & Guidelines for Results Files

This page defines standards and guidelines for QAP results files.

The target audience is script writers and maintainers of infrastructure.

General Philosophy

Writing a Test Script

Maintaining Infrastructure

Test Execution

See also How to Read a Results File.

See also Formal Grammar.


General Philosophy

Results files should attempt to attain the following goals: Every line in a results file should fall into one of the following categories:


Writing Test Scripts

If you are writing a test script, you are responsible for providing the following information in the results file: In general, your script should not log the following information:

Reporting Errors

Never use the built-in 4Test routine LogError!

When your script discovers an error condition, it should log the error in the results file in one of a number of different ways, depending on whose fault the error is:

Application
Errors
These are errors in the application. If your script or testcase can reasonably continue executing after discovering an error, then report it using this routine:
void CUlogError (string sErrorMessage);
If your script or testcase cannot reasonably continue executing after discovering an error, then report it by raising an exception with a 4Test raise statement.
Programming
Errors
These are errors made by the script writer (you), not by an application developer. Check for them using Assertions.
Setup
Errors
These are error made by the person running your script. Check for them and/or log them using these routines:
void RequireSetup (boolean bCondition, string sExplanation)
void CUsetupError (string sExplanation)

Composing Error Strings

Regardless of what mechanism you use to report an error, you should observe certain rules when you compose your error string:

Warnings

The use of warnings is reserved for indicating the use of obsolete or deprecated programming constructs. Warnings indicate a problem with a script (i.e. a programming error) which needs to be corrected eventually, but is not serious enough to prevent the script from being deployed.

Logging Exceptions

The only time your script should explicitly log an exception is when it explicitly catches the exception and does not reraise it. In all other cases, you should let the underlying infrastructure log the exception. Otherwise, the exception will be logged twice, leading to confusion and unnecessary clutter in the results file. In other words, if you catch an exception, you can reraise it or log it, but not both. (You can also recover from the exception and print nothing.)

If you must log an exception, always use the following routine:

void CUlogException (string sAdditionalInfo optional);

Output Data

Here we are referring to data that is generated directly by the script (i.e. through Print statements), not data that is generated by the application (i.e. bins, clips, EDLs, media files, etc.).

Output data must be clearly distinguished from other lines in the results file in one of the following ways:

For more information on test data, read this.

Functional Areas

Your script must report the functional areas that it tests. You must do this inside each testcase by calling CUtestcaseFeature (see below). If it is convenient, you can block out sections of your test that test a particular functional area by calling the following routines:

CUlogBlockState CUbeginFeatureTest (CUfeatureOrList)
void CUendFeatureTest (CUfeatureOrList)

For example, if you are testing the functional areas "Digitize | Functions | Batch Digitize | Drop" and "Digitize | Functions | Batch Digitize | Non-Drop", you can report the functional areas in one of two ways.

Example #1

In this example, the functional areas are completely defined inside the testcases.
testcase tcTestDrop ()
{
    CUtestcaseFeature({"Digitize","Functions","Batch Digitize","Drop"});

    ... body of test ...

}

testcase tcTestNonDrop ()
{
    CUtestcaseFeature({"Digitize","Functions","Batch Digitize","Non-Drop"});

    ... body of test ...

}

Example #2

In this example, the functional areas are partially defined inside the testcases, and are relative to the functional area defined in main.
testcase tcTestDrop ()
{
    CUtestcaseFeature("Drop");

    ... body of test ...

}

testcase tcTestNonDrop ()
{
    CUtestcaseFeature("Non-Drop");

    ... body of test ...

}

main ()
{
    CUbeginFeatureTest({"Digitize","Functions","Batch Digitize"});

    tcTestDrop();
    tcTestNonDrop();

    CUendFeatureTest({"Digitize","Functions","Batch Digitize"});
}

For more information on features and functional areas, click here.

Test Phases

A results file must properly report the phases of each test. The various test phases are:

1. eCU_TEST_SETUP Perform any setup that the test requires, such as opening bins, loading sequences, changing settings, etc. Some tests may not require any setup; some or all of a test's setup may be performed implicitly by the recovery system.
2. eCU_TEST_EXECUTE This phase should only include the operation being tested. Typically, this phase will consist of a single call to a subroutine or method. Every test must include this phase.
3. eCU_TEST_VERIFY Verify the results of the test, to ensure that the test was successful. Some verification may be performed implicitly by the infrastructure and/or the recovery system.
4. eCU_TEST_CLEANUP Perform any cleanup that the test requires, such as deleting sequences, closing dialogs, etc. You should normally leave around anything that the test created, so that it can be examined after the test script finishes running. Most tests will not require this phase.
5. eCU_TEST_RESULTS Display any results of the test. Most scripts will not use this phase; it is intended for scripts which first perform a set of tests, gathering data as they go along, and then print all the gathered data at the end.

Some of this reporting is taken care of for you by the Recovery System. Here is a summary:

In each testcase, you must explicitly set the test phase to eCU_TEST_EXECUTE immediately before executing the actual test. You may also set other test phase as needed. To set the test phase, call this routine:

void CUlogTestPhase (CUtestPhase ePhase);

Testcases

Within each testcase, you must report the functional area under test by calling CUtestcaseFeature. You should call it near the top of the body of your testcase.

Within each testcase, you must indicate the beginning of the actual test (as opposed to setup for the test) by calling CUlogTestPhase. If your testcase performs an cleanup after executing the test, you must indicate that with another call to CUlogTestPhase.

If your testcase takes parameters, then it must log information that distinguishes between different invocations of the testcase. Do this by placing a call to CUdisambiguateTC near the top of the body of the testcase.

Comments and Test Structure

It's a good idea to print occasional lines that describe, at a high level, what your test is trying to do. There are no special API's for such output; simply use the built-in 4Test functions Print and Printf.

You can begin any line with a period; such lines will never be interpreted to have special meaning.

Debugging Info

At some time during script development, you will probably want to print some information to the log file to help you debug the script. You should print such information by calling the following routine:

void CUdebugPrint (string sSubSystemPrefix, varargs of anytype);

You need not explicitly test a debugging flag before calling this routine; that functionality is built into CUdebugPrint.

Reserved Characters

Do not begin any lines of output with any of the following sequences of characters. They have special meaning to tools that read and interpret results files. You may use these sequences within a line, as long as they are not the first characters on the line.

asterisk
hyphen
less than
greater than
plus
colon
open bracket
close bracket
open paren.
close paren.
forward slash
backslash
exclamation
at
sharp
dollar
percent
circumflex
ampersand
underscore
equals
question mark
Script
Suite
Testcase

A line beginning with a period will never be interpreted to have special meaning.


Maintaining Infrastructure

The methods and routines of the infrastructure are responsible for the following:

Reporting GUI Actions

Report GUI actions by calling the following routine:

void CUlogGuiAction (string sAction, window w);

The action string should begin with a present-tense verb such as "click", "type", "select", or "close", and end in a preposition such as "in", "on", "to", or "from". If the verb takes both a direct and an indirect object, make sure that the one that isn't the window is included in the action string. For example, when reporting the selection of a menu item from a popup menu, include the name (or number) of the item in the action string.

The window should be the window, control, or pane that the action is being performed on. When calling CUlogGuiAction from a method, the window will almost always be "this". CUlogGuiAction automatically reports the window's parents.

Debugging Info

Print debugging information by calling the following routine:

void CUdebugPrint (string sSubSystemPrefix, varargs of anytype);

You need not explicitly test a debugging flag before calling this routine; that functionality is built into CUdebugPrint.


Test Execution

A few pieces of information must be provided by the person who is executing the test.

The name of the testbed must be provided in the LocalConstants.inc file by defining the following variable:

const string gsMC_TESTBED = "Lab System #1 (Moe)";

The internal release name of the product under test must be provided through the standard script parameter sMCinternalReleaseName. Examples are "Twister", "Jaguar", "Whirlpool".



This page is maintained by (REMOVED).

Last updated 12 May 2000.