CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data
Top Cited Papers
- 1 November 2006
- conference paper
- conference paper
- Published by Institute of Electrical and Electronics Engineers (IEEE)
Abstract
Emerging large-scale distributed storage systems are faced with the task of distributing petabytes of data among tens or hundreds of thousands of storage devices. Such systems must evenly distribute data and workload to efficiently utilize available resources and maximize system performance, while facilitating system growth and managing hardware failures. We have developed CRUSH, a scalable pseudorandom data distribution function designed for distributed object-based storage systems that efficiently maps data objects to storage devices without relying on a central directory. Because large systems are inherently dynamic, CRUSH is designed to facilitate the addition and removal of storage while minimizing unnecessary data movement. The algorithm accommodates a wide variety of data replication and reliability mechanisms and distributes data in terms of user-defined policies that enforce separation of replicas across failure domainsKeywords
This publication has 13 references indexed in Scilit:
- D-SPTFPublished by Association for Computing Machinery (ACM) ,2004
- FABPublished by Association for Computing Machinery (ACM) ,2004
- Replication under scalable hashing:a family of algorithms for scalable decentralized data distributionPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2004
- zFS - a scalable distributed file system using object disksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- Towards an object storePublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- SCADDAR: an efficient randomized technique to reorganize continuous media blocksPublished by Institute of Electrical and Electronics Engineers (IEEE) ,2003
- The Google file systemPublished by Association for Computing Machinery (ACM) ,2003
- Comparing random data allocation and data striping in multimedia serversPublished by Association for Computing Machinery (ACM) ,2000
- Consistent hashing and random treesPublished by Association for Computing Machinery (ACM) ,1997
- Efficiently extendible mappings for balanced data distributionAlgorithmica, 1996