HOG: Distributed Hadoop MapReduce on the Grid

Authors

Chen He, University of Nebraska-LincolnFollow
Derek J. Weitzel, University of Nebraska-LincolnFollow
David Swanson, University of Nebraska-LincolnFollow
Ying Lu, University of Nebraska-LincolnFollow

Date of this Version

11-10-2012

Citation

He, Chen, et al. "HOG: Distributed Hadoop MapReduce on the Grid." High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion:. IEEE, 2012.

Comments

© 2013 IEEE. Used by permission.

Abstract

MapReduce is a powerful data processing platform for commercial and academic applications. In this paper, we build a novel Hadoop MapReduce framework executed on the Open Science Grid which spans multiple institutions across the United States – Hadoop On the Grid (HOG). It is different from previous MapReduce platforms that run on dedicated environments like clusters or clouds. HOG provides a free, elastic, and dynamic MapReduce environment on the opportunistic resources of the grid. In HOG, we improve Hadoop’s fault tolerance for wide area data analysis by mapping data centers across the U.S. to virtual racks and creating multi-institution failure domains. Our modifications to the Hadoop framework are transparent to existing Hadoop MapReduce applications. In the evaluation, we successfully extend HOG to 1100 nodes on the grid. Additionally, we evaluate HOG with a simulated Facebook Hadoop MapReduce workload. We conclude that HOG’s rapid scalability can provide comparable performance to a dedicated Hadoop cluster.