Sunday, 8 July 2012
Amplifying Tests to Validate Exception Handling Code
P. Zhang, S. Elbaum, Amplifying Tests to Validate Exception Handling Code, in: International Conference on Software Engineering, IEEE, Zurich, Switzerland, 2012, pp. 606-616. http://bit.ly/MSY8EE
This is another Distinguished Paper from ICSE 2012. I am not a particular expert on testing and tend not to go out of my way to read testing papers that to my mind all too often comprise small twists on established techniques. This paper tackles an important problem: testing exception handling code for dealing with external resources. The examples in the paper include for example a geo-tagging app, a media centre controller, a barcode scanner, a cloud synced password keeper and a voIP client. The difficulty is the variety and complexity of the error behaviours of these resources. The paper provides an automated approach that 'systematically amplifies' the test suite by exploring the space of exceptional behaviour. The target program is instrumented adding a 'mocking device' in place of the external resources of interest, it then ampliﬁes a base test by exposing it to all possible exceptions thrown by the resource while monitoring for program failures. The approach generates mocking patterns which are pruned for length and then anomalies are filtered to generate a report that can be actioned. The approach is described in some detail with an architecture. It is evaluated and the paper reports that it can detect 65% of the faults reported in bug reports related to this class of problem and is precise enough that 77% of the anomalies detected correspond to faults ﬁxed by the developers. Take Home: The problem of using unreliable external resources, or incompletely understood APIs, can be partially addressed by thinking of it as a test coverage problem over the space of potential exceptional behaviour.