An Approach and Benchmark to Detect Behavioral Changes of Commits in Continuous Integration
share
When a developer pushes a change to an application's codebase, in the form of a commit, the newly introduced behavior may be incorrect. To prevent such regressions, developers rely on a continuous integration (CI) server to run a test suite on the application, at every commit. However, if the test suite lacks the test cases that specify the behavioral changes introduced by this commit, or the changes introduced by previous commits, the bug goes undetected. In this paper, we propose an approach that takes a program, its test suite, and a commit as input, and generates test methods that detect the behavioral changes of the commit, i.e., the behavioral difference between the pre-commit and post-commit versions of the program. In essence, this helps developers evolve the test suite (i.e., the application's specification) at the same time they evolve the application. We call our approach DCI (Detecting behavioral changes in CI). It works by generating variations of the existing test cases through (i) assertion amplification and (ii) a search-based exploration of input spaces. We evaluate our approach on a curated set of 50 commits from 5 open source Java projects. Our study exposes the characteristics of commits in modern open-source software and the ability of our approach to generate test methods that detect behavioral changes.
READ FULL TEXT