Search Based Testing for Code Coverage and Falsification in Cyber-Physical Systems

Abstract

State of the art test case generation tools vary in the way they exploit knowledge of the system under test. While experimental results demonstrate that such knowledge can improve the efficiency of the test case generation process, how to (automatically) retrieve and integrate such information in test generation is non-trivial. Especially, in the case of Cyber-Physical Systems (CPS), the problem is more challenging since CPS have not only software requirements but also physical functional requirements. In this work, we instrument a system's source code to extract run-time information related to the behavior of the system to determine the path taken through the source code itself. Given our focus on CPS, this information enables us to compute a metric representing the total coverage of the system behaviors for a series of inputs. We present the framework to instrument the code, and our method for using the instrumented code to calculate coverage-related robustness. We also demonstrate how to integrate this approach into a search to generate tests that can manage the trade-off between code coverage and identification of functionally unsafe behaviors (falsification). Our results show that our approach is more efficient than Uniform Random sampling in covering CPS operating modes (code). In addition, we show that our approach promotes better falsifications when violating inputs are present in a, possibly small, subset of the CPS states.