CSE Technical reports

VSFS: A Versatile Searchable File System for HPC Analytics

Lei Xu et al. — Wed, 01 May 2013 09:21:50 PDT

Big-data/HPC analytics applications have urgent needs for file-search services to drastically reduce the scale of the input data to accelerate analytics. Unfortunately, the existing solutions either are poorly scalable for large-scale systems, or lack well-integrated interface to allow applications to easily use them. We propose a distributed searchable file system, VSFS, which provide a novel and flexible POSIX-compatible searchable file system namespace that can be seamlessly integrate with any legacy code without modification. Additionally, to provide real-time indexing and searching performance, VSFS uses DRAM-based distributed consistent hashing ring to manages all file-index. The results of our evaluation show that VSFS is scalable in HPC environment. It achieves significant better file-indexing and file-searching performance than the popular SQL/NoSQL solutions, while it only introduces negligible overheads to raw I/O performance. Finally, we integrate the VSFS to a scientific analytic application to show its benefits in terms of performance and convenience.

A Scalable Inline Cluster Deduplication Framework for Big Data Protection

Yinjin Fu et al. — Thu, 19 Jul 2012 13:27:34 PDT

Cluster deduplication has become a widely deployed technology in data protection services for Big Data to satisfy the requirements of service level agreement (SLA). However, it remains a great challenge for cluster deduplica- tion to strike a sensible tradeoff between the conflicting goals of scalable dedu- plication throughput and high duplicate elimination ratio in cluster systems with low-end individual secondary storage nodes. We propose Σ-Dedupe, a scalable inline cluster deduplication framework, as a middleware deployable in cloud da- ta centers, to meet this challenge by exploiting data similarity and locality to op- timize cluster deduplication in inter-node and intra-node scenarios, respectively. Governed by a similarity-based stateful data routing scheme, Σ-Dedupe assigns similar data to the same backup server at the super-chunk granularity using a handprinting technique to maintain high cluster-deduplication efficiency with- out cross-node deduplication, and balances the workload of servers from backup clients. Meanwhile, Σ-Dedupe builds a similarity index over the traditional lo- cality-preserved caching design to alleviate the chunk index-lookup bottleneck in each node. Extensive evaluation of our Σ-Dedupe prototype against state-of- the-art schemes, driven by real-world datasets, demonstrates that Σ-Dedupe achieves a cluster-wide duplicate elimination ratio almost as high as the high- overhead and poorly scalable traditional stateful routing scheme but at an over- head only slightly higher than that of the scalable but low duplicate-elimination- ratio stateless routing approaches.

Solving the Search for Suitable Code: An Initial Implementation

Kathryn T. Stolee et al. — Fri, 29 Jun 2012 07:21:36 PDT

Searching for code is a common task among programmers, with the ultimate goal of finding and reusing code or getting ideas for implementation. While the process of searching for code - issuing a query and selecting a relevant match - is straightforward, several costs must be balanced, including the costs of specifying the query, examining the results to find desired code, and not finding a relevant result. For the popular syntactic searches the query cost is quite low, but the results are often vague or irrelevant, so the examination cost is high and matches may not be found. Semantic searches may return more relevant results, but current techniques that involve writing complex specifications or executing code against test cases are costly to the developer, and close matches cannot be easily identified. In this work, we address these limitations and propose an approach for semantic search in which developers specify lightweight, incomplete specifications and an SMT solver automatically identifies programs from a repository that match the specifications. The program repository is automatically encoded as constraints offline so the search for programs is efficient. The program encodings cover various levels of abstraction to enable partial matches when no, or few, exact matches exists. We present empirical evidence showing the lightweight specifications can be accurately defined by developers, instantiate this approach on a subset of the Yahoo! Pipes mashup language, and outline extensions to other programming languages.

A Generator for Random Non-Binary Finite Constraint Satisfaction Problems

Hui Zou et al. — Tue, 15 Nov 2011 13:54:39 PST

The paper describes an implementation of a generator of random instances of non-binary constraint satisfaction problems that meets a given set of specifications. This is a continuation of the work we started in Hui Zou, Amy M. Beckwith, and Berthe Y. Choueiry. A Generator of Random Instances of Binary Finite Constraint Satisfaction Problems with Controllable Levels of Interchangeability. Working note, Department of Computer Science and Engineering, University of Nebraska-Lincoln, 2001. July, 2001.

The program is designed to generate random instances of Constraint Satisfaction Problems (CSPs) that meet a set of specified parameters, such as the number of variables, domain size, constraint density, tightness. At the same time, it can generate any combination of binary, ternary, and/or quaternary constraints specifies as percentage of the total number of constraints in the problem.

Relational Neighborhood Inverse Consistency for Constraint Satisfaction

Robert J. Woodward et al. — Tue, 15 Nov 2011 13:51:14 PST

Freuder and Elfe [1996] introduced Neighborhood Inverse Consistency (NIC) as a new local consistency property for Constraint Satisfaction Problems (CSPs) that filters the domains of variables. Two advantages of the algorithm for enforcing NIC is that it automatically adapts its filtering power to the local connectivity of the network and has insignificant space overhead. In this document, we discuss Relational Neighborhood Inverse Consistency (RNIC), which is an extension of NIC to filter relations introduced in [Woodward et al., 2011a], how we enhance the propagation effectiveness by reformulating the dual graph of the CSP. We also describe an automated selection policy that outperforms all approaches in a statistically significant manner.

Tree-Based Algorithms for Computing k-Combinations and k-Compositions

Shant Karakashian et al. — Tue, 15 Nov 2011 13:47:33 PST

In this document, we describe two tree-based algorithms for computing all k-combinations and k-compositions of a finite set.

We have developed two algorithms for solving the following combinatorial tasks:

• Given a finite set S and a natural number k, find all subsets of S of size k. In the literature, this problem is called k-subsets and k-combinations.

• Given two natural numbers k, n where k [less than or equal to] n, find all k-compositions of n where a k-composition is an ordered combination of k nonzero natural numbers whose sum is n. Note that in the literature, a k-composition of n can have null numbers. Further, some authors require that the sum of the k numbers to be less or equal to n. Both algorithms are based on building an intermediary tree data-structure. Using similar tree structures for generating various combinatorial objects under constraints is a “reasonably standard approach” [Hartke 2010]. Algorithms exist in the literature for k-combinations and k-compositions. For example, Wilf [1989] discusses combinatorial Gray codes, and attributes an algorithm for k-compositions to Knuth. Ruskey [1993] shows a bijection between the compositions of Knuth and the combinations of Eades and McKay [1984]. Algorithm pseudocode for those combinatorial problems is reported in Google1, Section 4.3 and 5.7 of [Ruskey 2010], and [Arndt 2010a; 2010b]. The goal of this document is to report the pseudocode of the algorithms implemented in our software [Karakashian 2010].

Flocking Over 3D Terrain

Joel Gompert — Tue, 15 Nov 2011 13:44:20 PST

A method is presented for animating herds of animals that can follow terrain while being efficient enough to run in real-time. This method involves making simple modifications to Reynolds’ agent-based flocking algorithm. The modifications use only local properties of the terrain, and thus have low complexity. This method focuses on using terrain that can be described as an elevation grid, but it may be extendible to arbitrary terrain. The flocking algorithm with these modifications produces naturally behaving herds that follow the terrain. They will swerve around hills and attempt to follow paths that reduce energy expenditure. The terrain-following rule added to the flocking algorithm has a constant parameter that can be adjusted to produce different behaviors. Empirical analysis shows how this parameter affects energy expenditure of the animals while traveling over the terrain.

Combining Ordering Heuristics and Bundling Techniques for Solving Finite Constraint Satisfaction Problems

Amy Beckwith et al. — Tue, 15 Nov 2011 13:37:35 PST

We investigate techniques to enhance the performance of backtrack search procedure with forward-checking (FC-BT) for finding all solutions to a finite Constraint Satisfaction Problem (CSP). We consider ordering heuristics for variables and/or values and bundling techniques based on the computation of interchangeability. While the former methods allow us to traverse the search space more effectively, the latter allow us to reduce it size. We design and compare strategies that combine static and dynamic versions of these two approaches. We show empirically the utility of dynamic variable ordering combined with dynamic bundling in both random problems and puzzles.

Isomorph-free generation of 2-connected graphs with applications

Derrick Stolee — Fri, 19 Aug 2011 07:19:49 PDT

Many interesting graph families contain only 2-connected graphs, which have ear decompositions. We develop a technique to generate families of unlabeled 2-connected graphs using ear augmentations and apply this technique to two problems. In the first application, we search for uniquely Kr-saturated graphs and find the list of uniquely K4-saturated graphs on at most 12 vertices, supporting current conjectures for this problem. In the second application, we verify the Edge Reconstruction Conjecture for all 2-connected graphs on at most 12 vertices. This technique can be easily extended to more problems concerning 2-connected graphs.

Propeller: A Scalable Metadata Organization for A Versatile Searchable File System

Lei Xu et al. — Thu, 21 Jul 2011 07:35:11 PDT

The exponentially increasing amount of data in file systems has made it increasingly important for users, administrators and applications to be able to fast retrieve files using file-search services, instead of replying on the standard file system API to traverse the hierarchical namespaces. The quality of the file-search services is significantly affected by the file-indexing overhead, the file-search performance and the accuracy of search results. Unfortunately, the existing file-search solutions either are so poorly scalable that their performance degrades unacceptably when the systems scale up, or incur so much crawling delays that they produce acceptably inaccurate results. We believe that the time is ripe for the re-designing of a searchable file system capable of accurate and scalable system-level file search.

The main challenge facing the design and implemention of such a searchable file system is how to update file indices in \emph{real-time} in a scalable way to obtain accurate file-search results. Updating file indices in hierarchical file systems or existing file-search solutions usually induces performance bottleneck and limits scalability. Thus we propose a lightweight, scalable and metadata organization, \emph{Propeller}, for future searchable file systems. Propeller partitions the namespace according to file-access patterns, which exposes massive parallelism for the emerging manycore architecture, and provides versatile system-level file-search functionalities, to support future searchable file systems. The extensive evaluation results of our \emph{Propeller} prototype show that it achieves significantly better file-index and file-search performance than a database-based solution (MySQL) and only incurs negligible overhead to the normal file I/O operations on a state-of-the-art file system (Ext4).

Rapport: Semantic-sensitive Namespace Management in Large-scale File Systems

Yu Hua et al. — Tue, 08 Jun 2010 06:30:36 PDT

Explosive growth in volume and complexity of data exacerbates the key challenge to effectively and efficiently manage data in a way that fundamentally improves the ease and efficacy of their use. Existing large-scale file systems rely on hierarchically structured namespace that leads to severe performance bottlenecks and renders it impossible to support real-time queries on multi-dimensional attributes. This paper proposes a novel semantic-sensitive scheme, called Rapport, to provide dynamic and adaptive namespace management and support complex queries. The basic idea is to build files’ namespace by utilizing their semantic correlation and exploiting dynamic evolution of attributes to support namespace management. Extensive tracedriven experiments validate the effectiveness and efficiency of our proposed schemes. To the best of our knowledge, this is the first work on semantic-sensitive namespace management for ultra-scale file systems.

DSFS: Decentralized Security for Large Parallel File Systems

Zhongying Niu et al. — Sat, 27 Mar 2010 19:27:21 PDT

This paper describes DSFS, a decentralized security system for large parallel file system. DSFS stores global access control lists (ACLs) in a centralized decisionmaking server and pushes pre-authorization lists (PALs) into storage devices. Thus DSFS allows users to flexibly set any access control policy for the global ACL or even change the global ACL system without having to upgrade the security code in their storage devices. With pre-authorization lists, DSFS enables a networkattached storage device to immediately authorize I/O, instead of demanding a client to acquire an authorization from a centralized authorization server at a crucial time. The client needs to acquire only an identity key from an authentication server to access any devices she wants. Experimental results show that DSFS achieves higher performance and scalability than traditional capability-based security protocols.

COSET: Cooperative Set Last Level Caches

Dongyuan Zhan et al. — Tue, 16 Feb 2010 11:11:59 PST

The speed gap between processors and DRAM remains a crit-ical performance bottleneck for contemporary computer systems, which necessitates an effective management of last level caches (LLC) to minimize expensive off-chip accesses. However, because all sets in a conventional set-associative cache design are statically assigned an equal number of blocks, the LLC capacity utilization can drastically diminish when the cache actually exhibits non-uniform capacity demands across the sets. To reveal the wide exis-tence of set-level non-uniformity of capacity demand in real appli-cations, this technical report first establishes an accurate metric for measuring individual sets’ capacity demands by developing a group of mathematical models. Then, the report presents a last-level cache design1 called COSET (COoperative SET) L2 cache that identifies the capacity needs of individual sets based on the new metric, dynamically couples two sets with complementary capacity demands, and enables the set with a higher resource de-mand to utilize the capacity of its coupled set to reduce conflict misses. Our simulation study on 6 selected SPEC CPU 2000 benchmarks shows that the COSET L2 cache achieves a MPKI of as low as 0.383 and 0.781 on average normalized to the standard LRU cache, outperforming the state-of-the-art approach SBC that has the best and average performance results of 0.585 and 0.867 respectively.

Design of An All-Optical WDM Lightpath Concentrator

Shivashis Saha et al. — Wed, 13 Jan 2010 07:57:28 PST

A design of a nonblocking, all-optical lightpath concentrator using WOC and WDM crossbar switches is presented. The proposed concentrator is highly scalable, cost-efficient, and can switch signals in both space and wavelength domains without requiring a separate wavelength conversion stage.

Selection of Switching Sites in All-Optical Nework Topology Design

Shivashis Saha et al. — Thu, 07 Jan 2010 14:38:12 PST

In this paper, we consider the problem of topology design for optical networks. We investigate the problem of selecting switching sites to minimize total cost of the optical network. The cost of an optical network can be expressed as a sum of three main factors: the site cost, the link cost, and the switch cost. To the best of our knowledge, this problem has not been studied in its general form as investigated in this paper.

We present a mixed integer quadratic programming (MIQP) formulation of the problem to find the optimal value of the total network cost. We also present an efficient heuristic to approximate the solution in polynomial time.

The experimental results show good performance of the heuristic. The value of the total network cost computed by the heuristic varies within 2% to 21% of its optimal value in the experiments with 10 nodes. The total network cost computed by the heuristic for 51% of the experiments with 10 node network topologies varies within 8% of its optimal value. We also discuss the insight gained from our experiments.

Exploiting Set-Level Non-Uniformity of Capacity Demand to Enhance CMP Cooperative Caching

Dongyuan Zhan et al. — Mon, 09 Nov 2009 09:58:52 PST

As the Memory Wall remains a bottleneck for Chip Multiprocessors (CMP), the effective management of CMP last level caches becomes of paramount importance in minimizing expensive off-chip memory accesses. For the CMPs with private last level caches, Cooperative Caching (CC) has been proposed to enable capacity sharing among private caches by spilling an evicted block from one cache to another. But this eviction-driven CC does not necessarily promote cache performance since it implicitly favors the applications full of block evictions regardless of their real capacity demand. The recent Dynamic Spill-Receive (DSR) paradigm improves cooperative caching by prioritizing applications with higher benefit from extra capacity in spilling blocks. However, the DSR paradigm only exploits the coarse-grained application-level difference in capacity demand, making it less effective as the non-uniformity exists at a much finer level.

This paper (i) highlights the observation of cache set-level non-uniformity of capacity demand, and (ii) presents a novel L2 cache design, named SNUG (Set-level Non-Uniformity identifier and Grouper), to exploit the fine-grained non-uniformity to further enhance the effectiveness of cooperative caching. By utilizing a per-set shadow tag array and saturating counter, SNUG can identify whether a set should either spill or receive blocks; by using an index-bit flipping scheme, SNUG can group peer sets for spilling and receiving in an flexible way, capturing more opportunities for cooperative caching. We evaluate our design through extensive execution-driven simulations on Quald-core CMP systems. Our results show that for 6 classes of workload combinations our SNUG cache can improve the CMP throughput by up to 22.3%, with an average of 13.9%, over the baseline configuration, while the state-of-the-art DSR scheme can only achieve an improvement by up to 14.5% and 8.4% on average.

Refactoring Pipe-like Mashups for End-User Programmers

Kathryn T. Stolee et al. — Fri, 11 Sep 2009 23:46:58 PDT

Mashups are becoming increasingly popular as end users are able to easily access, manipulate, and compose data from many web sources. We have observed, however, that mashups tend to suffer from deficiencies that propagate as mashups are reused. To address these deficiencies, we would like to bring some of the benefits of software engineering techniques to the end users creating these programs. In this work, we focus on identifying code smells indicative of the deficiencies we observed in web mashups programmed in the popular Yahoo! Pipes environment. Through an empirical study, we explore the impact of those smells on end-user programmers and observe that users generally prefer mashups without smells. We then introduce refactorings targeting those smells, reducing the complexity of the mashup programs, increasing their abstraction, updating broken data sources and dated components, and standardizing their structures to fit the community development patterns. Our assessment of a large sample of mashups shows that smells are present in 81% of them and that the proposed refactorings can reduce the number of smelly mashups to 16%, illustrating the potential of refactoring to support the thousands of end users programming mashups.

An Analysis of MCMC Sampling Methods for Estimating Weighted Sums in Winnow

Qingping Tao et al. — Fri, 14 Aug 2009 10:10:15 PDT

Chawla et al. introduced a way to use the Markov chain Monte Carlo method to estimate weighted sums in multiplicative weight update algorithms when the number of inputs is exponential. But their algorithm still required extensive simulation of the Markov chain in order to get accurate estimates of the weighted sums. We propose an optimized version of Chawla et al.’s algorithm, which produces exactly the same classifications while often using fewer Markov chain simulations. We also apply three other sampling techniques and empirically compare them with Chawla et al.’sMetropolis sampler to determine how effective each is in drawing good samples in the least amount of time, in terms of accuracy of weighted sum estimates and in terms of Winnow’s prediction accuracy.

Technical Reports (1999 - 2005)

Fri, 14 Aug 2009 10:10:14 PDT

Authors of Technical Reports (1999-2005):
Choueiry, Berthe
Elbaum, Sebastian
Goddard, Steve
Henninger, Scott
Jiang, Hong
Rothermel, Gregg
Scott, Stephen
Seth, Sharad
Soh, Leen-Kiat
Variyam, Vinodchandran

Technical Reports (2004 - 2009)

Fri, 14 Aug 2009 10:10:13 PDT

Authors of Technical Reports (2005-2009):
Choueiry, Berthe
Cohen, Myra
Deogun, Jitender
Dwyer, Matthew
Elbaum, Sebastian
Goddard, Steve
Henninger, Scott
Jiang, Hong
Lu, Ying
Ramamurthy, Byrav
Rothermel, Gregg
Scott, Stephen
Seth, Sharad
Soh, Leen-Kiat
Srisa-an, Witty
Swanson, David
Variyam, Vinodchandran
Wang, Jun
Xu, Lisong