TPTP Testing Strategy Part 2

Test Conventions and Best Practices

 

1.0 TPTP Test Project Conventions

               1.1 Naming Conventions

               1.2 Test Project Detailed Structure

2.0 Test Creation Best Practices

               2.1 Creating Manual Test Suites

               2.2 Creating JUnit and JUnit Plug-in Test Suites

               2.3 Creating AGR Test Suites

3.0 Test Execution Best Practices

               3.1 Smoke Testing

4.0 TPTP Test Project Exceptions

               4.1 JVMPI Profiler Test Suites

5.0 The Common Test Infrastructure

               5.1 Reference Platform

               5.2 Adding Tests

               5.3 Installing and Deploying

 

 

1.0 TPTP Test Project Conventions

1.1 Naming conventions:

  1. All names for testing resources and folder names MUST adhere to the following rules:
  2. Names must NOT contain any white space characters (for example, tab, space, and new line).
  3. Every name should begin with an upper or lower case letter (for example, A-Z or a-z). All other characters in the name should be upper or lower case letters, digits, or the underscore.
  4. Java keywords (for example, import, if, etc.) cannot be used for names of JUnit test resources.
  5. The name of the Deployment file (deployment files are often used for remote launch of test suites) may contain '.' characters. The name should be meaningful to the project and the host that it is associated with. For example, a local deployment file for the org.eclipse.hyades.use.cases folder can be org.eclipse.hyades.use.cases.localhost.deploy
  6. The name and directory structure containing the generated execution history file MUST be the same as it will be stored in CVS. If the generated execution history file is either moved or renamed, the linkage with the associated test suite will break thereby corrupting generated reports. In the event of an existing execution history in the same directory, TPTP Testing Tools will append a millisecond time stamp to the name of the new execution history file.
  7. Test suite names must start with the bugzilla component name (see bugzilla 104409) associated with the plug-in or component under test.
  8. Multiple JRE or platform test suite names must end with the name, vendor, and version of the JRE or platform under test. These test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.

1.2 Test Project Detailed Structure

 

The contents of a test project must follow a defined structure. Table 1 details the required structure for contents of a test project, for example, test-results/platform/org.eclipse.hyades.use.cases.

Name

Purpose

deployment

Directory containing all deployment configurations for executing the test suites in the test project.

gui

Directory containing all AGR test suites for the test project.

gui/AllTests.testsuite

AGR test suite to execute all lower level AGR test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite (see Table 2).

gui/AllSmokeTests.testsuite

AGR smoke test suite to execute all lower level AGR smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllGUISmokeTests.testsuite (see Table 2).

gui_results

Directory containing all execution histories for automated GUI test suites for the test project.

junit_plugin

Directory containing all JUnit Plug-in test suites for the test project.

junit_plugin/AllTests.testsuite

JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite (see Table 2).

junit_plugin/AllSmokeTests.testsuite

JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginSmokeTests.testsuite (see Table 2).

junit_plugin_results

Directory containing all execution histories for JUnit Plug-in test suites for the test project.

junit

Directory containing all JUnit test suites for the test project.

junit/AllTests.testsuite

JUnit test suite to execute all lower level JUnit test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite (see Table 2).

junit/AllSmokeTests.testsuite

JUnit smoke test suite to execute all lower level JUnit smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllJUnitSmokeTests.testsuite (see Table 2).

junit_results

Directory containing all execution histories for JUnit test suites for the test project.

manual

Directory containing all manual test suites for the test project.

manual/AllTests.testsuite

Manual test suite to execute all lower level manual test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite (see Table 2).

manual/AllSmokeTests.testsuite

Manual smoke test suite to execute all lower level manual smoke test suites for the test project. Must be referenced by test-results/platform/org.eclipse.hyades.tests/AllManualSmokeTests.testsuite (see Table 2).

manual_results

Directory containing all execution histories for manual test suites for the test project.

resources

Directory containing any resources required to execute the test suites for the test project.

src

Directory containing generated JUnit Java source code for the JUnit and JUnit Plug-in test suites and Java source code corresponding to verification hooks of AGR test suites for the test project.

Table 1: Required structure for contents of a test project.

Each of the subfolders contains a test project named org.eclipse.hyades.use.cases for testing functionality that crosses plug-in or component boundaries.

The test-results/platform/org.eclipse.hyades.tests test project contains all root-level test suites for TPTP. Table 2 details the required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.

Name

Purpose

deployment

Directory containing all deployment configurations for executing the root-level test suites in TPTP.

enablement

Directory containing all the TPTP enablement tests.

TVT

Directory containing all the TPTP Translation Verification Tests (TVT).

src

Directory containing generated JUnit Java source code for the root-level JUnit and JUnit Plug-in test suites in TPTP and Java source code corresponding to verification hooks of root-level AGR test suites in TPTP.

AllGUITests.testsuite

Root-level AGR test suite to execute all lower level AGR test suites.

AllGUISmokeTests.testsuite

Root-level AGR smoke test suite to execute all lower level AGR smoke test suites.

AllJUnitPluginTests.testsuite

Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites.

AllJUnitPluginSmokeTests.testsuite

Root-level JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test suites.

AllJUnitTests.testsuite

Root-level JUnit test suite to execute all lower level JUnit test suites.

AllJUnitSmokeTests.testsuite

Root-level JUnit smoke test suite to execute all lower level JUnit smoke test suites.

AllManualTests.testsuite

Root-level manual test suite to execute all lower level manual test suites.

AllManualSmokeTests.testsuite

Root-level manual smoke test suite to execute all lower level manual smoke test suites.

Table 2: Required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.

 

The execution history files (*.execution) generated when executing test suites are checked-in to the CVS branch for the release under test by the committer for the associated plug-in or component. For example, the execution history files generated from testing the current full release would be checked in to HEAD. Execution history files are checked in to the *_results folder for the type of test suite. For example, execution history files generated from executing a manual test suite would be checked in the manual_results folder. To delineate between test executions on multiple platforms, each *_results folder contains the following subfolders for each supported platform:

·         AIX

·         AS_400

·         HP_UX

·         Linux_390

·         Linux_x86

·         OS_390

·         Solaris

·         Windows

·         Linux PPC64

Supported platform subfolders may be optionally appended with the platform version and release:

·         AIX_5_2_0

·         AS_400_5_2

·         HP_UX_11

·         Linux_390_SuSE_ Enterprise_7_0

·         Linux_x86_RedHat_Enterprise_2_1

·         Linux_x86_SLES_10

·         OS_390_1_6

·         Solaris_9

·         Windows_2000

·         Windows_XP

·         Windows_Vista

·         Linux_PPC64

Test execution reports are generated and checked in to CVS by the TPTP Project Lead or PMC representative at predetermined milestones in a test pass for public web access. Test execution reports are available at: www.eclipse.org/tptp >> Development Plans >> Test Pass Reports

2.0 Test Creation Best Practices

 

Based on the previous experiences of test resource creators and testers, the following general recommendations will assist testers on creating test cases. Section 2.1 - 2.3 go into more details about best practices for creating manual, JUnit, JUnit Plug-in, and AGR test suites.

  1. A test case should always have a specific purpose for testing a scenario. Having many test cases in a test suite does not necessarily ensure better quality. Each test case must be well thought out and have a specific purpose. Simply testing the basic functionality of a tool is often not enough to ensure high quality standards. Boundary cases are usually considered to be interesting test cases. As an example, consider the case of testing a function that removes entries from a table. The following test cases should be considered:
  2. Avoid repeating the same set of steps in multiple test cases. If two test cases require the user to import a log file, then create three test cases. The first test case will only serve as setting the environment for successive test cases and the second and third test case will test the specific functionality in mind.
  3. This point provides a basic summary of the first two: apply the high cohesion principle to test cases. Cohesion means assigning specific responsibilities to test cases. A test case requiring a user to import a log file, profile an application, and use the Probekit has little common in its responsibilities and thus does not follow the high cohesion principle.
  4. Avoid unnecessarily complicating test cases that are designed to test functionality and not performance. For example, a test case that uses a class with 3 methods to test the Java Profiler is likely as good as a test case that uses a class with 10 methods.

 

               2.1 Creating Manual Test Suites

1.      Create small and terse manual test cases that cover well-defined partitions of functionality. Smaller sized test cases provide easier units of work for one tester to execute, more realistic reporting statistics and concise steps to reproduce defects.

2.      Create manual test suites with sufficient test cases for one tester to complete in less than an hour. Smaller sized test suites provide easier units of work dividing test suites between testers while decreasing the risk of lost execution data.

3.      In the event of manual test suites being executed by multiple testers, have the tester(s) tabulate the execution histories on paper and the tester with largest assigned test cases will record all the execution histories in the TPTP Manual Test Client.

4.      When structuring the internal organization as the associated test suite folder (for example, manual), create a summary test suite for the subfolder (for example,/manual/AllTests.testsuite). The summary test suite (for example,/manual/AllTests.testsuite) will reference all test suites within the current folder. This summary test suites is registered with the root-level manual test suite (for example, test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.

5.      Although the TPTP Manual Test Client permits recording more than one execution verdict per test case, it is recommended to only record one execution verdict. Otherwise, TPTP Testing Tools arbitrates the execution verdicts based on the following precedence rules (decreasing order) which may cause unintended execution histories:

o        Error

o        Fail

o        Inconclusive

o        Pass

 

               2.2 Creating JUnit and JUnit Plug-in Test Suites

 

1.      Create small and terse unit test cases that cover individual methods or sequence of methods. Smaller sized test cases provide more realistic reporting statistics and concise steps to reproduce defects.

2.      When structuring the internal organization as the associated test suite folders (for example, junit and junit_plugin), create summary test suites for each subfolder (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite). The summary test suites (for example,/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite) will reference all test suites within the current folder. These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites (for example, test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite and test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.

               2.3 Creating AGR Test Suites

 

   The user is expected to have reviewed the available user guide document on the AGR before reading the recommendations below. The user guide is available on the document section linked from TPTP's web site.

1.      Ensure that a recording session is short. Avoid having a generated macro for a test case that is longer than 100 lines.

2.      As mentioned previously make sure that the set of test cases included in a test suite don't perform redundant operations. For example, if two test cases require launching a process, then launch the process once and use it for each of the test cases that follow. At the same time the test cases shouldn't be too dependent on the environment setup by previous test cases. A test case should easily be executable in quick mode without having to run many test cases that precede it. It becomes difficult to debug a test case when it is dependent on more than one test case that precedes it.

3.      Avoid using absolute paths in a test case. Absolute paths should be replaced with relative paths using one of the available static variables (for example, %testsuiteProjectLocation%). Other modifiable fields that are dependent on the environment that the test suite is launched in (for example, host name) should be linked to a datapool variable.

4.      Use a logger to make it convenient to debug verification hooks. See test-results/platform/ org.eclipse.hyades.use.cases/src/org.eclipse.hyades.use.cases.auto.common/Logger.java

5.      Comment the test case thoroughly in the description section of the test case. Include all manual steps that the test case is performing.

6.      If any resource (such as a datapool) needs to be modified before the test suite is executed, then include that as part of the description of the test suite. Any other important information that the tester should be aware of needs to be included in the description of the test suite.

7.      If a manual test case is automated, then add '(DEPRECATED - Use <automated test suite name>)' to the beginning of the description of the manual test case.

8.      Avoid position-based recording as much as possible. If position-based recording must be used for a component, then try to include all such test cases of the component in a separate test suite. Clearly note the resolution that the tester should use as part of the description of the test suite.

9.      Verification hooks should verify the value of controls rather than their logical representation. For example, if a tree item represents object A, then verify the existence of the tree item by testing that its text field is the same as the name of object A as opposed to just simply testing for the existence of object A.

There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:

1.      The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. When structuring the internal organization as the associated test suite folder (for example, gui), create a summary test suite for the subfolder (for example,/gui/AllTests.testsuite). The summary test suite (for example,/gui/AllTests.testsuite) will reference all test suites within the current folder. This summary test suites is registered with the root-level gui test suite (for example, test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite), thereby reducing work when creating and removing plug-in or component test suites.

2.      The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated.

3.      A resource directory should be used to contain the resources required for the verification hooks of a test suite. The name of the directory should be <test-suite-name>.Resources (For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. This directory may contain any number of log files or any other files that the test suite depends on). If a datapool is being associated with a test suite, then include the datapool as part of this directory (see the user guide document on the AGR for more details).

4.      Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites

5.      See figure 1 as an example of how the 'gui' folder can be structured.


Figure 1: The structure of the 'gui' folder of org.eclipse.tptp.monitoring.tests

3.0 Test Execution Best Practices

 

               3.1 Smoke Testing

1.      Smoke testing, or patch/maintenance release testing, refers to partial regression test for testing the majority of the functionality in less time than a full test pass.

2.      Smoke test suites should cover:

o        All major use cases.

o        All major supported platforms.

o        All major supported JREs.

o        All automated test suites (for example, JUnit, JUnit Plug-in, and AGR test suites).

3.      Smoke test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.

4.0 TPTP Test Project Exceptions

 

Some TPTP tests are not manual, AGR or TPTP JUnit/JUnit Plug-in tests as described above.

 

               4.1 JVMPI Profiler Test Suites

 

   The JVMTI Profiler tests (Platform.Agents.JVMPI.*) are run like regular JUnit tests but require manual configuration of the local host and installation and configuration of a test server on   the remote host. This is documented in test-results/platform/org.eclipse.tptp.ac.testautomation/automation-files/notes/ACTestAutomation.doc

 

 

5.0 The Common Test Infrastructure

 

The common test infrastructure is used for build verification test.

 

               5.1 Reference Platform

              

               TPTP will use reference platforms. New code will be expected to be tested against the reference platform before being committed.

               The build verification test will run on the reference platform.

 

               The current reference platform for TPTP is: IBM Java 1.5 (latest SR) and Windows XP/x86.

              

               5.2 Adding Tests

 

               The BVT test suites are:

               org.eclipse.hyades.tests\BVT\AllBVTGUITests.testsuite

               org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests.testsuite

               org.eclipse.hyades.tests\BVT\AllBVTJUnitTests.testsuite

 

               5.3 Installing and Executing

              

               Please refer to the BVT Readme at org.eclipse.tptp.platform.releng.tools/testautomation/readme.html