Extending DIRAC file management with erasure-coding for efficient storage

Skipsey, S. C. , Todev, P., Britton, D. , Crooks, D. and Roy, G. (2015) Extending DIRAC file management with erasure-coding for efficient storage. Journal of Physics: Conference Series, 664(4), 042051. (doi: 10.1088/1742-6596/664/4/042051)

124424.pdf - Published Version
Available under License Creative Commons Attribution.



The state of the art in Grid style data management is to achieve increased resilience of data via multiple complete replicas of data files across multiple storage endpoints. While this is effective, it is not the most space-efficient approach to resilience, especially when the reliability of individual storage endpoints is sufficiently high that only a few will be inactive at any point in time. We report on work performed as part of GridPP[1], extending the Dirac File Catalogue and file management interface to allow the placement of erasure-coded files: each file distributed as N identically-sized chunks of data striped across a vector of storage endpoints, encoded such that any M chunks can be lost and the original file can be reconstructed. The tools developed are transparent to the user, and, as well as allowing up and downloading of data to Grid storage, also provide the possibility of parallelising access across all of the distributed chunks at once, improving data transfer and IO performance. We expect this approach to be of most interest to smaller VOs, who have tighter bounds on the storage available to them, but larger (WLCG) VOs may be interested as their total data increases during Run 2. We provide an analysis of the costs and benefits of the approach, along with future development and implementation plans in this area. In general, overheads for multiple file transfers provide the largest issue for competitiveness of this approach at present.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Crooks, Dr David and Roy, Dr Gareth and Britton, Professor David and Skipsey, Dr Samuel
Authors: Skipsey, S. C., Todev, P., Britton, D., Crooks, D., and Roy, G.
Subjects:Q Science > QC Physics
College/School:College of Science and Engineering > School of Physics and Astronomy
Journal Name:Journal of Physics: Conference Series
Publisher:IOP Publishing
ISSN (Online):1742-6596
Published Online:23 December 2015
Copyright Holders:Copyright © 2015 The Authors
First Published:First published in Journal of Physics: Conference Series 664(4): 042051
Publisher Policy:Reproduced under a Creative Commons License

University Staff: Request a correction | Enlighten Editors: Update this record