TPTP Testing Strategy Part 1

Executing and Creating TPTP tests

 

 

1.0 Preparation

  1.1 Install the TPTP Testing Tools

  1.2 Extracting the Test Resources

  1.3 Structure of the Test Resources

2.0 Creating Tests

  2.1 Manual Test Suites

  2.2 JUnit and JUnit Plug-in Test Suites

  2.3 AGR Test Suites

3.0 Executing Tests

  3.1 Manual Test Suites

  3.2 JUnit and JUnit Plug-in Test Suites

  3.3 AGR test Suites

4.0 Common Tasks

  4.1 Creating Test Suites

  4.2 Launching Test Suites

  4.3 Viewing Test Logs

  4.4 Report Test Results

5.0 Help

 

­1.0 Preparation

 

1.1 Install the TPTP testing tools

 

  1. Open the TPTP download site.
  2. Select the release of TPTP to be tested and download and unzip the TPTP plugins as well as the requirements listed in the “TPTP Plugins for Eclipse” section.
  3. Download and unzip Automated GUI Recording (“As Is” section).
    1. Note: Remember to restart eclipse with the –clean option after installing the AGR
  4. If remote execution is required download and unzip the Agent Controller (“Agent Controller” section).
  5. The Installation Guide and Release Notes are located in the “Documentation” section.

 

1.2 Extracting the Test Resources

 

 

  1. Check out the individual test projects under test-results/monitoring/, test-results/platform/, test-results/test/, or test-results/trace/ to the local workspace.

For Example: test-results/platform/org.eclipse.hyades.use.cases

  1. To ensure that the local workspace contains the latest test projects, update at the beginning of each test pass and report generation.

 

1.3 Structure of the Test Resources

 

  1. Test projects must follow a defined structure. Please see Part 2 (link needed) for the required structure of test project contents. When structuring the internal organization of the associated test suite folders (for example, junit and junit_plugin), create summary test suites for each subfolder (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite). The summary test suites (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite) will reference all test suites within the current folder. These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites (test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite and test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.

 

AGR tests structure

  1. There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:
    1. The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. When structuring the internal organization of the associated test suite folder (for example, gui), create a summary test suite for the subfolder (for example,/gui/AllTests.testsuite). The summary test suite (for example,/gui/AllTests.testsuite) will reference all test suites within the current folder. This summary test suites is registered with the root-level gui test suite (test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.
    2. The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated (if applicable).
    3. A resource directory should be used to contain the resources required for the verification hooks of a test suite. The name of the directory should be <test-suite-name>.Resources (For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. This directory may contain any number of log files or any other files that the test suite depends on). If a datapool is being associated with a test suite, then include the datapool as part of this directory
    4. Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites

 

2.0 Creating New TPTP Tests

 

To create new TPTP test suites and cases please follow the documentation in the TPTP Help and AGR documentation.

For guidelines specific to testing TPTP see the Common Tasks (need link) section below.

 

2.1 Creating Manual Test Cases and Suites

 

  1. For TPTP 4.5.0 and greater download and install the Manual Test Tools from the As-Is Components section of the TPTP download page.
  2. Detailed steps for creating manual test cases and suites can be found under Help >> Help Contents >> Testing Applications >> Testing Manually
  3. Manual tests are created in the manual folder of a test project. Please refer to the naming conventions section of Part 2 (need link).
  4. The TPTP Manual Test Case Generator may be used to generate the structure of manual test case descriptions based on the standardized structured manual test case format.
  5. Try to create suites with a number of cases that can be completed in less than one hour.

 

 

2.2 Creating Junit and Junit Plug-in Test Cases and Suites

 

  1. JUnit tests are created in the JUnit folder of a test project, Junit Plug-ins tests in the junit_plugin folder. Please refer to the naming conventions section of Part 2 (need link).
  2. Detailed steps for creating new TPTP JUnit tests can be found under Help >> Help Contents >> Testing Applications >> Testing with JUnit
  3. Detailed steps for creating new TPTP JUnit Plug-in tests can be found under Help >> Help Contents >> Testing Applications >> Testing plug-ins with JUnit

 

2.3 Creating AGR Test Suites

 

  1. For TPTP 4.5.0 and greater download the Automated GUI Recorder from the As-Is Components section of the TPTP download page.

2.   Detailed steps for creating AGR test cases and suites can be found in the document “An Introduction to Using TPTP's Automated GUI Recorder”  found at http://www.eclipse.org/tptp/test/documents/index.php

 

 

3.0 Executing Tests

 

3.1 Manual Test Suites

 

  1. Open the Test perspective.
  2. In the Test Navigator view, select the manual test suite to be executed.
  3. Launch the selected test suite by creating a Test configuration in the Run dialog

Under the Test tab select the test suite

Choose the results folder associated with the selected test suite and deployment platform

Note: The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.

Click Run

  1. Launch a candidate workbench on the testing platform
  2. Execute each test case as specified in the TPTP Manual Test View on the local host.
  3. Record the result of the test case execution by capturing the status of the test case execution using the following explanation for TPTP Testing Tools Status options:

TPTP Testing Tools Status

Explanation

Error

Do not use.

Fail

The test case could be executed and failed due to a blocking defect.

Inconclusive

The test case was not applicable OR the test case could be executed and failed due to a non-blocking defect.

A non-blocking defect is defined as a nice-to-have fix that does not degrade the core functionality being tested by the test case.

Pass

The test case could be executed and passed.

 

 

 

 

 

 

 

 

 

·        For verdicts of Fail or Inconclusive please provide a defect number and explanation in the test log.

·        For instructions on submitting defects from the test log which will create a link from the test log to bugzilla please see Submitting Defects (need link) in the Common tasks (need link) section.

 

 

 

3.2 JUnit and JUnit Plug-in Test Suites

 

  1. Open the Test perspective.
  2. In the Test Navigator view, select the JUnit test suite to be executed.
  3. Launch the selected test suite by creating a Test configuration in the Run dialog

Under the Test tab select the test suite

Select the deployment.

To run a test remotely you will need to create and configure a new test deployment

For JUnit plug-in tests you also may want to specify deployment on a different workbench location.

The steps to create a new location and deployment are in the TPTP Project Documentation under Testing Plugins with JUnit

Choose the results folder associated with the selected test suite and deployment platform

Note: The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.

Click Run

  1. An execution history will be generated according to the following explanation of TPTP Testing Tools Status options: 

TPTP Testing Tools Status

Explanation

Error

The test case was or could not be executed due to a run time error.

Fail

The test case could be executed and failed due to a defect.

Pass

The test case could be executed and passed.

 

 

 

 

 

 

 

·        The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.

·        For instructions on submitting defects from the test log which will create a link from the test log to bugzilla please see Submitting Defects (need link) in the Common tasks (need link) section.

 

 

3.3 AGR Test Suites       

  1. Ensure that the Agent Controller is installed and started.
  2. Open the Test perspective.
  3. In the Test Navigator view, select the Automated GUI Test to be executed.
  4. Execute the test suite by creating a launch configuration item. See Creating Launch Configurations (need link) in the Common Tasks (need link) section of this document.

Important: Using the 'Quick Run' mode will not generate an execution history file. Make sure that a proper launch configuration is used.

  1.  The runner will generate an execution history according to the following explanation of TPTP Testing Tools Status options:

TPTP Testing Tools Status

Explanation

Error

The test case was or could not be executed due to a run time error.

Fail

The test case could be executed and failed due to a defect.

Pass

The test case could be executed and passed.

 

 

 

 

 

 

 

·        The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.

·        For instructions on submitting defects from the test log which will create a link from the test log to bugzilla please see Submitting Defects (need link) in the Common tasks (need link) section.

 

 

 

 

4.0 Common Tasks

 

 

4.1 Creating Test Suites

 

General Guidelines for test case creation:

  1. Create short concise cases that cover identifiable partitions of functionality. This increases reporting accuracy and allows for easy reproduction by testers and fixers.
  2. Comment test cases thoroughly in the description section of the including the manual steps for what the test case is performing.
  3. If any resource (such as a datapool) needs to be modified before the test suite is executed, then include that as part of the description of the test suite. Any other important information that the tester should be aware of needs to be included in the description of the test suite.
  4. If an existing manual test case is being automated, then add '(DEPRECATED - Use <automated test suite name>)' to the beginning of the description of the manual test case.
  5. For more guidance creating effective test cases please refer to the best practices section of Part 2 (need link).

 

4.2 Launching Test Suites

 

Creating Test Deployments

            Detailed steps for creating test deployments can be found in Help >> Help Contents >> Testing Applications >> Creating a Test Deployment

 

Create a Test Launch Configuration

  1. Create a new launch configuration by selecting the entry Test from the list of configurations in Run >> Run Configurations
  2. Use the “Select test to run” pane to navigate to and select the test suite you wish to run.
  3. Select the deployment for running the test suite. By default the test will be deployed locally; to change this see the Creating Test Deployments (need link) topic. Manual tests are always run in the local deployment.
  4. In the Test Logs tab override the default log configuration to match the desired log directory (for example plugin_results). Please refer to the naming conventions section of Part 2 (need link) for the names of result directories.

 

4.3 Viewing Test Logs

To open a test log file, use the Open File window, or double-click in the Test Navigator the file with the extension, .execution.

  1. From the File menu, click Open File.
  2. Browse to and select the test log file to open. (Test log files use the extension, .execution.)
  3. Click OK.

4.4 Reporting Test Results

 

Submitting Execution Histories

 

  1. Execution histories should be checked-in to the CVS branch for the release under test.  They are checked into a results folder under the associated plugin for the type of test executed (manual_results, junit_results, junit_plugin_results, gui_results). To delineate between test executions on multiple platforms, each results folder contains subfolders for the supported platforms.

For Example: The results of executing the manual test org.eclipse.hyades.use.cases/manual/Profiling_and_Logging/Platform.Communication.Control_Channel_Remote_Profiling.Windows_IA32.testsuite should be generated to and checked into org.eclipse.hyades.use.cases/manual_results/Profiling_and_Logging/Windows_XP

  1. Testers who are not committers should email the execution history files to the committer for the associated component.

Include the following in your email:

Name of tester if not same as sender

Deployment platform(s) including name, version and release.

Execution specific information including name, version and release.

TPTP driver build id and test pass identifier.

 

5.0 Help

 

Eclipse Test & Performance Tools Platform Project Web Site

 

Full current TPTP Test Tools Documentation can be found in the eclipse help under:  Help >> Help Contents >> Testing Applications

 

AGR Documentation

            http://www.eclipse.org/tptp/test/documents/index.php

            See “An Introduction to Using TPTP's Automated GUI Recorder” for your TPTP release version.

 

Documentation for TPTP extenders

 

TPTP Mailing Lists and Newsgroups