APIs
     
Subsystem
list
     
Top
level

Guidelines for Using Test Data

This document discusses some issues regarding the use of test data, and offers some guidelines and standards for same.

Where Should Input Data Live?
A Standard Location for External Input Data
Golden Samples
Media Files
Where Should Output Data Live?
A Standard Location for Output Data
What About Results from Previous Tests?
Golden Samples
Standard Projects and Bins
Rules for Using Standard Projects
Rules for Using Standard Bins
The Default Project and Bin


Where Should Input Data Live?

By "Input Data", we mean data that is used as an input to a piece of test code. We assume that without its input data, such test code is unusable. Therefore, maintaining the integrity of the input data is just as important as maintaining the integrity of the test script code. And so, test data should be stored under source control (ClearCase). However, this does not adequately answer the question of where it should live.

There are basic choices of where test data can reside: within a 4Test file (part of the test script), or in an external file. (Data can also be generated on the fly at runtime, but such data is out of the scope of this discussion, and ultimately is generated from other data, even if the starting point is merely a seed for a random number generator.) Here are some pros and cons regarding these choices:

Data Lives in Test ScriptData Lives in External File
The test script must be modified in order to modify the test data. The test data can be modified independently of the test script.
The syntax of the test data can be checked at compile time. The syntax of the test data must be checked at run time.
The data is used primarily by the test script, and is not read directly by the application. The data may be read by the test script or by the application (or both).

Some types of data, such as Projects, Bins and Media Files, cannot reasonably be stored in a 4Test file, and must reside in files external to the test script. Furthermore, some such files must live in locations dictated by the operating system or the application.

A Standard Location for External Input Data

In MClocal.inc there is a constant called glsMC_INPUT_DATA_DIR that contains a pathname to a directory in which all external input data files should reside. This path will normally point into the ClearCase directory "/ppg/QAP_Canned_Data". This directory contains a sub-directory for each subsystem whose scripts require external input data. Input data files that are used by a variety of subsystems live in a directory called "Shared".

Golden Samples

A "golden sample" is an output file that has been examined and determined to satisfy some set of requirements. It is used for comparing against a future output file to make sure that the new output file is correct. Golden samples are used when it is not feasible for a test script to examine an output file and analyze it for correctness.

Note that while golden samples were originally output files, for the purposes of a test script, they function as input files: they are read by the script (or some auxiliary comparison program), not generated by the script, or by the application during script execution. Therefore, golden samples should be stored along with other external input data.

Media Files

Because of their size, we do not currently encourage checking them into ClearCase. A solution to this issue needs to be found.


Where Should Output Data Live?

By "Output Data", we mean data that is generated by the test script or the application as a result of executing the script.

If output data is generated directly by the test script, there are two basic choices of where it can reside: within the QA Partner results file (using Print, printf, etc.), or in a separate file.

If output data is generated by the application, the application may (or may not) dictate where the data resides.

A Standard Location for Output Data

In MClocal.inc there is a constant called glsMC_OUTPUT_DATA_DIR that contains a pathname to a directory in which all output data files should reside, except for the QA Partner results file, and files whose location is dictated by the application.

The output directory should contain a sub-directory for each subsystem whose scripts generate output data files. That sub-directory should be created by the scripts as needed.

What About Results from Previous Test Runs?

The user of a test script may (or may not) want to retain the results of previous test runs. The designer of a test script has two choices:

Golden Samples

Golden samples are treated as input files. Read this.


Standard Projects and Bins

  • Standard Data Main Page Additional notes are here.


    The Default Project and Bin

    In MClocal.inc there is a pair of constants called gsMC_DEFAULT_PROJECT and gsMC_DEFAULT_BIN. These constants define a project and bin which may be used by any test script.

    Requirements for Content

    The default project is required to contain at least one bin (which must be the default bin), and the bin is required to contain at least one simple source clip. There is no guarantee that there is media associated with any clips. The default project and bin may or may not be one of the standard test projects and bins.

    Rules for Usage

    If a test script uses the default project or bin, it can make no assumptions about their contents other than what is stated above. In particular, you should not assume that the default project is one of the standard projects, nor should you assume that the default bin is one of the standard bins.



    This page is maintained by (REMOVED).

    Last updated 29 January 1998.