Holland Computing Center
Date of this Version
Summer 7-27-2015
Citation
Weitzel, D., Bockelman, B. & Swanson, D. Distributed Caching Using the HTCondor CacheD. In Proceedings for Conference on Parallel and Distributed Processing Techniques and Applications, 2015.
Abstract
A batch processing job in a distributed system has three clear steps, stage-in, execution, and stage-out. As data sizes have increased, the stage-in time has also increased. In order to optimize stage-in time for shared inputs, we propose the CacheD, a caching mechanism for high throughput computing. Along with caching on worker nodes for rapid transfers, we also introduce a novel transfer method to distribute shared caches to multiple worker nodes utilizing BitTorrent. We show that our caching method significantly improves workflow completion times by minimizing stage-in time while being non-intrusive to the computational resources, allowing for opportunistic resources to utilize this caching method.