On Adequacy of Assertions in Automated Test Suites: An Empirical Investigation

An integral part of test case is the verification phase (also called 'test oracle'), which verifies program's state, output or behavior. In automated testing, the verification phase is often implemented using test assertions which are usually developed manually by testers. More precisely, assertions are used for checking the unit or the system's behavior (or output) which is reflected by the changes in the data fields of the class under test, or the output of the function under test. Originated from human (testers') error, test suites are prone to having inadequate assertions. The paper reports an empirical study on the Inadequate-Assertion (IA) problem in the context of automated test suites developed for open-source projects. In this study, test suites of three active open-source projects have been chosen. To investigate IA problem occurrence among the sampled test suites, we performed mutation analysis and coverage analysis. The results indicate that: (1) the IA problem is common among the sampled open-source projects, and the occurrence varies from project to project and from package to package, and (2) the occurrence rate of the IA problem is positively co-related with the complexity of test code.