A framework for statistical testing of software components

Abstract

Statistical testing involves the testing of software by selecting test cases from a probability distribution that is intended to represent the software's operational usage. In this paper, we describe and evaluate a framework for statistical testing of software components that incorporates test case execution and output evaluation. An operational profile and a test oracle are essential for the statistical testing of software components because they are used for test case generation and output evaluation respectively. An operational profile is a set of input events and their associated probabilities of occurrence expected in actual operation. A test oracle is a mechanism that is used to check the results of test cases. We present four types of operational profiles and three types of test oracles, and empirically evaluate them using the framework by applying them to two software components. The results show that while simple operational profiles may be effective for some components, more sophisticated profiles are needed for others. For the components that we tested, the fault-detecting effectiveness of the test oracles was similar.