Human Performance Regression Testing

Authors

Amanda Swearngin, University of Nebraska-LincolnFollow
Myra B. Cohen, University of Nebraska-LincolnFollow
Bonnie E. John, IBM T. J. Watson Research CenterFollow
Rachel K. E. Bellamy, IBM T. J. Watson Research CenterFollow

Date of this Version

2013

Citation

Software Engineering (ICSE), 2013 35th International Conference on Year: 2013 Pages: 152 - 161, DOI: 10.1109/ICSE.2013.6606561

Comments

Copyright 2013 IEEE. Used by permission.

Abstract

As software systems evolve, new interface features such as keyboard shortcuts and toolbars are introduced. While it is common to regression test the new features for functional correctness, there has been less focus on systematic regression testing for usability, due to the effort and time involved in human studies. Cognitive modeling tools such as CogTool provide some help by computing predictions of user performance, but they still require manual effort to describe the user interface and tasks, limiting regression testing efforts. In recent work, we developed CogTool-Helper to reduce the effort required to generate human performance models of existing systems. We build on this work by providing task specific test case generation and present our vision for human performance regression testing (HPRT) that generates large numbers of test cases and evaluates a range of human performance predictions for the same task. We examine the feasibility of HPRT on four tasks in LibreOffice, find several regressions, and then discuss how a project team could use this information. We also illustrate that we can increase efficiency with sampling by leveraging an inference algorithm. Samples that take approximately 50% of the runtime lose at most 10% of the performance predictions.