Computer Science and Engineering, Department of
Date of this Version
Summer 8-16-2013
Document Type
Article
Citation
Zhang, P. (2013). Automated test case generation to validate non-functional software requirements. (Doctoral dissertation, University of Nebraska-Lincoln).
Abstract
A software system is bounded by a set of requirements. Functional requirements describe what the system must do, in terms of inputs, behavior, and outputs. We define non-functional requirements to be how well these functional requirements are satisfied, in terms of qualities or constraints on the design or on the implementation of a system. In practice, the validation of these kinds of requirements, does not receive equal emphasis. Techniques for validating functional requirements target all levels of software testing phases, and explore both black-box and white-box approaches. Techniques for validating non-functional requirements, on the other hand, largely operate in a black-box manner, and focus mostly on system testing level. As a result, in practice more efforts are focused on validating functional requirements, and only assess non-functional requirements after functional validation is complete.
In this dissertation, we propose a set of automated bounded exhaustive white-box testing techniques that enable cost-effective validation of non-functional requirements from two perspectives. For non-functional requirements defined as qualities of a system, we target load testing for the purpose of performance validation. We present Symbolic Load Generation (SLG), a load test suite generation approach that uses symbolic execution to exhaustively traverse program execution paths, and produce test cases for the ones that may expose worst-case or near worst-case resource consumption scenarios. An assessment of SLG on a set of Java applications shows that it generates test suites that induce program response times and memory consumption several times worse than the compared alternatives, it scales to large and complex inputs, and it exposes a diversity of resource consuming program behavior. We subsequently present CompSLG, a compositional load test generation technique that aims to achieve better scalability than SLG. CompSLG is fully automated to handle software that follows the pipeline architecture, and is evaluated on a group of Unix and XML pipelines. The results show that it generates load tests in situations where SLG fails to scale, and achieves comparable load with a fraction of the cost. We also extend CompSLG to enable the handling of more complex software structures in terms of Java programs, and show the viability of the extended technique through a series of examples.
For non-functional requirements defined as constraints on a system, we target validation of contextual constraints that are imposed by external resources with which the software interacts. We first assessed the magnitude of the problem by conducting a study on fault repositories of several popular Android applications. We then present an approach that amplifies existing tests in an exhaustive manner to validate exception handling constructs that are used to handle such constraints. Our assessment of the approach on a set of Android mobile applications indicates that it can be fully automated, is powerful enough to detect 67% of the faults reported in the bug reports of this kind, and is precise enough that 78% of the detected anomalies correspond to faults fixed by the developers.
Advisor: Sebastian Elbaum
Combined, the two proposed techniques advance the field of automated software testing by providing white-box support for non-functional validation.
Comments
A DISSERTATION Presented to the Faculty of The Graduate College at the University of Nebraska In Partial Fulfillment of Requirements For the Degree of Doctor of Philosophy, Major: Computer Science, Under the Supervision of Professor Sebastian Elbaum. Lincoln, Nebraska: August, 2013
Copyright (c) 2013 Pingyu Zhang