Run your system tests as often as possible. Ideally, run your tests whenever new code is checked into your version control tool.

For larger systems, this will be too expensive. Your tests may take several hours to run; therefore, you'll need to schedule tests less frequently. If possible, however, run the tests several times a day. As a minimum, run automated tests each night, but also run them during working hours, if possible.

Run the test at the scheduled time based on the instructions in the Test Script. It is best that the script be automated.

Good practices:

1. Run the test in a separate test environment.
2. Ensure that you run the test against the latest clean build.
3. The first step of the test should be to set up the test environment (ensure that the network is available, that the test database is available and reset to a known state, and so on).

1. Close the test logs. The appropriate test log files should be closed and placed in the appropriate folder or directory.
2. Announce results. Send a notice to everyone involved in the project, informing them of the result of the test run and where they can find the test logs.

Collect and compile information from test execution logs so you can:

• Capture the high-impact and risk issues discovered in running the tests.
• Identify errors in test creation, data inputs, or integrating applications and any architectural anomalies.
• Isolate the target of the test to determine failure points.
• Diagnose failure symptoms and characteristics.
• Assess and identify possible solutions.

After completing these steps, verify that you have enough details to determine the impact of the results. In addition, make sure that enough information exists to assist individuals who are performing dependent tasks.

Identify whether or not the test has failed and propose a solution based on the type of test and category of failure.  The test The approach to testing will determine the identified failures and candidates for solutions.

Tests that are programmer supporting are used to help prepare and ensure confidence in the code. When identifying failures and proposing solutions for programmer supporting tests:

• Failures will be identified at an object or element level
• Solutions will be to help clarify the problem

Test that are business supporting are used to uncover prior mistakes and omissions.

• Failures will identify omissions in requirements
• Solutions will help to clarify expectations of the system

After you have this information and the steps proposed to resolve the failures, you can effectively categorize the type of failure and the appropriate type of solution.

See [MAR03] for more information.

Communicate the test results to the team. For failed tests this might involve adding bugs to the Work Items List.

Communicating test results can affect the perception of the effectiveness of the tests. When communicating test results, it is important that you:

• Know the audience, so that appropriate information is communicated appropriately
• Run tests or scenarios that are likely to uncover the high-impact and risk issues or represent actual use of the system

When preparing test result reports, answer the following questions:

• How many test cases exist, and what are their states (pass, fail, blocked, and so on)?
• How many bug reports have been filed, and what are their states (open, assigned, ready for testing, closed, deferred)?
• What trends and patterns do you see in test case and bug report states, especially opened and closed bug reports and passed and failed test cases?
• For test cases that were blocked or skipped, why are they in this state?
• Considering all test cases not yet run (and perhaps not even created yet), what key risks and areas of functionality remain untested?
• For failed test cases, what are the associated bug reports?
• For bug reports ready for confirmation testing, when can your team perform the test?