Discovery Over Application: A Case Study of Misaligned Incentives in Software Engineering

Authors

Eric F. Rizzi, University of Nebraska-LincolnFollow

Date of this Version

Summer 7-31-2015

Citation

Rizzi, Eric F. Discovery Over Application: A Case Study of Misaligned Incentives in Software Engineering. MS Thesis. University of Nebraska-Lincoln Lincoln, NE, 2015.

Comments

A THESIS Presented to the Faculty of The Graduate College at the University of Nebraska In Partial Fulfillment of Requirements For the Degree of Master of Science, Major: Computer Science, Under the Supervision of Matthew B. Dwyer and Sebastian Elbaum. Lincoln, Nebraska: August, 2015

Copyright (c) 2015 Eric F. Rizzi

Abstract

In this thesis, we present evidence that there is an under-emphasis on the application of software systems in Software Engineering research, affecting the advancement of the field as a whole. Specifically, we perform a case-study on KLEE, a tool with over 1000 citations. We made improvements that consisted of fixing performance bugs and implementing optimizations that have become common practice, increasing KLEE's performance by 2-11X. To understand how techniques proposed in the literature would be affected by these improvements, we analyzed 100 papers that cited the original KLEE paper. From this analysis we found two things. First, it is clear that coherence to the principles of replication is lacking; it was often very difficult to understand how a particular study used KLEE, and therefore to understand how our improvements would affect the study. Second, when conservatively estimating how the studies relied on KLEE, we believe that seven of the 21 papers that we investigated could have their conclusions significantly strengthened or weakened. Upon closer investigation, six of these seven papers involved studies that directly compared a KLEE or a KLEE dependent tool to some other tool. The potential for mis-application within these competing techniques makes it difficult to understand which observations are true, a situation that potentially leads to wasted effort and slowed progress. To conclude, we examine several recent proposals to address this under-emphasis, using KLEE as an exemplar to understand their likely effects.

Advisers: Matthew B. Dwyer and Sebastian Elbaum