• Developer Test

This guideline describes a number of types of tests.  To perform these types of testing you need to define, and then run, a series of tests against the source code. A developer test is a single test that needs to be performed.

It is valuable to augment automated tests with human readable test scripts in order to implement developer test cases, scripts that include the information discussed below. A test script is the actual steps, sometimes either written procedures to follow or the source code of a test. Developer test scripts are run against testing targets: either one unit of source code, a more complex portion of your system (such as a component), or the entire system itself to test some developer issue such as integration.

Regression Testing

Regression testing is the act of ensuring that changes to the code have not adversely affected existing functionality. It is important to recognize that incremental development makes regression testing critical. Whenever you release an application, you must ensure its previous functionality still works, and because you release applications more often when taking the incremental approach, this means regression testing becomes that much more important. Regression testing is the first thing you should be thinking about when testing for the following reasons:

1. You want to be able to modify code and know that you can rerun your tests to see if you broke anything.
2. Existing users get very angry when things that previously worked don’t work anymore.

Regression testing is fairly straightforward conceptually – you just need to run all of the previous test cases against the new version of the code. Regression testing tools help immensely because they are designed with regression testing in mind. However, there are potential challenges to regression testing:

• When you change your production, either to enhance it or to refactor it, you will need to rework existing test cases coupled to that code.
• If your updates affect only a component of the system, then potentially you only need to run the test cases that affect this single component. Although this approach is a little risky because your changes may have had a greater impact than you suspect, it does help to reduce both the time and cost of regression testing.
• The more non-code artifacts that you decide to keep, the greater the effort to regression test your work and therefore the greater the risk to your project because you are more likely to skimp on your testing efforts.

Regression testing is critical to success as an agile developer. Many software developers use the xUnit family of open source tools, such as JUnit and VBUnit, to test their code. The advantage of these tools is that they implement a testing framework with which you can regression test all of your source code. Commercial testing tools are also viable options.

Traditional Code Testing Techniques

Although object and procedural technologies are different, several important testing concepts from the procedural world are valid regardless of the underlying technology. These traditional testing techniques are:

• Black-box testing
• Clear-box testing
• Boundary-value testing
• Coverage/Path testing

Black-Box Testing

Black-box testing, also called interface testing, is a technique in which you create test cases based only on the expected functionality of a method, class, or application without any knowledge of its internal workings. One way to define black-box testing is that given input A you should obtain expected results B.

The goal of black-box testing is to ensure the system can do what it should be able to do, but not how it does it. For example, if you invoke differenceInDays(June 30 2006, July 3 2006) the expected result should be three. The creation of black-box tests is often driven by the requirements for the system. The basic idea is to look at the user requirement and ask what needs to be done to show that the user requirement is met.

The primary advantage of black-box testing is that it enables you to prove that your application fulfills the requirements defined for it.   The primary disadvantage is that it does not show that the internals of the system work (hence the need for clear-box testing).

Clear-Box Testing

Clear-box testing, also called white-box testing, is based on the idea that the program code can drive the development of test cases. The basic concept is you look at the code, and then create test cases that exercise it. For example, assume you have access to the source code for differenceInDays(). When you look at it, you see an IF statement determines whether the two dates are in the same year. If they are in the same year then a simple strategy based on Julian dates is used; if not then a more complex strategy is used. This indicates that you need at least one test that uses dates from the same year and one from different years. By looking at the code, you are able to determine new test cases to exercise the different logic paths within it.

The primary advantage of this concept is that it motivates you to create tests that exercise specific lines of code.  The disadvantages are that it does not ensure that your code fulfils the actual requirements (hence the need for black-box testing) and that your testing code becomes highly coupled to your application code.

Boundary-Value Testing

This is based on the knowledge that you need to test your code to ensure it can handle unusual and extreme situations. For example, boundary-value test cases differenceInDays() would include passing it the same date, two wildly different dates, one date on the last day of the year and the second on the first day of the following year, and one date on February 29th of a leap year. The basic idea is you want to look for limits defined either by your business rules or by common sense, and then create test cases to test attribute values in and around those values.

The primary advantage of boundary value testing is that it motivates you to confirm that your program code is able to handle “unusual” or “extreme” cases.

Coverage and Path Testing

Two critical “traditional” concepts are coverage and path testing. Coverage testing is a technique in which you create a series of test cases designed to test all the code paths in your code. In many ways, coverage testing is simply a collection of clear-box test cases that together exercise every line of code in your application at least once. Path testing is a superset of coverage testing that ensures not only have all lines of code been tested, but all paths of logic have also been tested. The main difference occurs when you have a method with more than one set of case statements or nested IF statements: to determine the number of test cases with coverage testing you would count the maximum number of paths between the sets of case/nested IF statements and, with path testing, you would multiply the number of logic paths.

Unit and Integration Testing

Unit testing is the testing of an item, such as an operation, in isolation. For example, the tests defined so far for differenceInDays() are all unit tests. Integration testing, on the other hand, is the testing of a collection of items to validate that they work together. In the case of the data library/class, do the various functions work together? Perhaps the differenceInDays() function has a side effect that causes the dayOfWeek() function to fail if differenceInDays() is called first. Integration testing looks for problems like this.

Object-Oriented Testing Techniques

When testing systems built using object technology it is important to understand that your source code is composed of several constructs, including methods (operations), classes, and inheritance. Therefore you need testing techniques that reflect the fact that you have these constructs. These techniques are:

• Method testing
• Class testing
• Class-integration testing
• Inheritance-regression testing

Method Testing

Method testing is the act of ensuring that your methods (operations) perform as defined. In procedural testing this would have been called function/procedure testing. Issues to address via method testing include:

• Ensure that your getter/setter methods work as intended
• Ensure that each method returns the proper values, including error messages and exceptions
• Validate the parameters being passed to a method
• Ensure that a method does what it should

The advantage of method testing is that it ensures that methods work well in isolation but it does not help to find unintended side effects.

Class Testing

The main purpose of class testing is to test classes in isolation, and it is effectively the combination of traditional unit testing and integration testing. It is unit testing because you are testing the class and its instances as single units in isolation, but it is also integration testing because you need to verify the methods and attributes of the class work together. The one assumption you need to make while writing “class tests” is that all other classes in the system work. Issues to address via class testing include:

• Validate that the attributes of an object are initialized properly
• Validate the invariants of the class

The primary advantages of class testing are that it validates that the operations and properties of a class work together and that the class works in isolation. However, it does not guarantee that a class works well with the rest of your system.

Class-integration Testing

Also known as component testing, this technique addresses the issue of whether the classes in your system, or a component of your system, work together properly. The relationships between classes can be used to drive the development of class integration test cases. Issues to address via class-integration testing include:

• Validate that objects do what other objects expect of them
• Validate that return values are acted upon appropriately
• Validate that exceptions/errors are processed appropriately

The technique helps to validate that the various classes within a component, or a system, work together. However, it can be difficult to define and develop the test cases to fully perform this level of testing.

Inheritance-regression Testing