Research and Innovation, UNL Office of
Holland Computing Center: Faculty Publications
Accessibility Remediation
If you are unable to use this item in its current form due to accessibility barriers, you may request remediation through our remediation request form.
Document Type
Article
Date of this Version
Summer 7-27-2015
Citation
Weitzel, D., Bockelman, B. & Swanson, D. Distributed Caching Using the HTCondor CacheD. In Proceedings for Conference on Parallel and Distributed Processing Techniques and Applications, 2015.
Abstract
A batch processing job in a distributed system has three clear steps, stage-in, execution, and stage-out. As data sizes have increased, the stage-in time has also increased. In order to optimize stage-in time for shared inputs, we propose the CacheD, a caching mechanism for high throughput computing. Along with caching on worker nodes for rapid transfers, we also introduce a novel transfer method to distribute shared caches to multiple worker nodes utilizing BitTorrent. We show that our caching method significantly improves workflow completion times by minimizing stage-in time while being non-intrusive to the computational resources, allowing for opportunistic resources to utilize this caching method.