Experimenting with Software Testbeds for Evaluating New Technologies

Authors

Mikael Lindvall, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Ioana Rus, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Paolo Donzelli, University of Maryland, College ParkFollow
Atif Memon, University of Maryland, College ParkFollow
Marvin Zelkowitz, University of Maryland, College ParkFollow
Aysu Betin-Can, Middle East Technical University, Ankara, TurkeyFollow
Tevfik Bultan, University of California at Santa Barbara,Follow
Chris Ackermann, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Bettina Anders, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Sima Asgari, University of Maryland, College ParkFollow
Victor Basili, University of Maryland, College ParkFollow
Lorin Hochstein, University of Nebraska-LincolnFollow
Jörg Fellmann, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Forrest Shull, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Roseanne Tvedt, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Daniel Pech, Fraunhofer Center for Experimental Software Engineering, College Park, MDFollow
Daniel Hirschbach, University of Maryland, College ParkFollow

Document Type

Article

Date of this Version

2007

Comments

Published in Empirical Software Engineering 12 (2007), pp. 417-444; doi: 10.1007/s10664-006-9034-0 Copyright © 2007 Springer Science + Business Media, LLC. Used by permission.

Abstract

The evolution of a new technology depends upon a good theoretical basis for developing the technology, as well as upon its experimental validation. In order to provide for this experimentation, we have investigated the creation of a software testbed and the feasibility of using the same testbed for experimenting with a broad set of technologies. The testbed is a set of programs, data, and supporting documentation that allows researchers to test their new technology on a standard software platform. An important component of this testbed is the Unified Model of Dependability (UMD), which was used to elicit dependability requirements for the testbed software. With a collection of seeded faults and known issues of the target system, we are able to determine if a new technology is adept at uncovering defects or providing other aids proposed by its developers. In this paper, we present the Tactical Separation Assisted Flight Environment (TSAFE) testbed environment for which we modeled and evaluated dependability requirements and defined faults to be seeded for experimentation. We describe two completed experiments that we conducted on the testbed. The first experiment studies a technology that identifies architectural violations and evaluates its ability to detect the violations. The second experiment studies model checking as part of design for verification. We conclude by describing ongoing experimental work studying testing, using the same testbed. Our conclusion is that even though these three experiments are very different in terms of the studied technology, using and re-using the same testbed is beneficial and cost effective.