Deep Neural Network Test Coverage: How Far Are We?
share
DNN testing is one of the most effective methods to guarantee the quality of DNN. In DNN testing, many test coverage metrics have been proposed to measure test effectiveness, including structural coverage and non-structural coverage (which are classified according to whether considering which structural elements are covered during testing). Those test coverage metrics are proposed based on the assumption: they are correlated with test effectiveness (i.e., the generation of adversarial test inputs or the error-revealing capability of test inputs in DNN testing studies). However, it is still unknown whether the assumption is tenable. In this work, we conducted the first extensive study to systematically validate the assumption by controlling for the size of test sets. In the study, we studied seven typical test coverage metrics based on 9 pairs of datasets and models with great diversity (including four pairs that have never been used to evaluate these test coverage metrics before). The results demonstrate that the assumption fails for structural coverage in general but holds for non-structural coverage on more than half of subjects, indicating that measuring the difference of DNN behaviors between test inputs and training data is more promising than measuring which structural elements are covered by test inputs for measuring test effectiveness. Even so, the current non-structural coverage metrics still can be improved from several aspects such as unfriendly parameters and unstable performance. That indicates that although a lot of test coverage metrics have been proposed before, there is still a lot of room for improvement of measuring test effectiveness in DNN testing, and our study has pointed out some promising directions.
READ FULL TEXT