Analysis and improvement of data-set level file distribution in Disk Pool Manager

Skipsey, S. , Purdie, S., Britton, D. , Mitchell, M., Bhimji, W. and Smith, D. (2014) Analysis and improvement of data-set level file distribution in Disk Pool Manager. In: International Conference on Computing in High Energy and Nuclear Physics (CHEP 2013), Amsterdam, The Netherlands, 14-18 Oct 2013, 042042. (doi: 10.1088/1742-6596/513/4/042042)

Abstract

Of the three most widely used implementations of the WLCG Storage Element specification, Disk Pool Manager[1, 2] (DPM) has the simplest implementation of file placement balancing (StoRM doesn't attempt this, leaving it up to the underlying filesystem, which can be very sophisticated in itself). DPM uses a round-robin algorithm (with optional filesystem weighting), for placing files across filesystems and servers. This does a reasonable job of evenly distributing files across the storage array provided to it. However, it does not offer any guarantees of the evenness of distribution of that subset of files associated with a given "dataset" (which often maps onto a "directory" in the DPM namespace (DPNS)). It is useful to consider a concept of "balance", where an optimally balanced set of files indicates that the files are distributed evenly across all of the pool nodes. The best case performance of the round robin algorithm is to maintain balance, it has no mechanism to improve balance.<p></p> In the past year or more, larger DPM sites have noticed load spikes on individual disk servers, and suspected that these were exacerbated by excesses of files from popular datasets on those servers. We present here a software tool which analyses file distribution for all datasets in a DPM SE, providing a measure of the poorness of file location in this context. Further, the tool provides a list of file movement actions which will improve dataset-level file distribution, and can action those file movements itself. We present results of such an analysis on the UKI-SCOTGRID-GLASGOW Production DPM.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Skipsey, Dr Samuel and Mitchell, Mr Mark and Purdie, Dr Stuart and Britton, Professor David
• Authors: Skipsey, S., Purdie, S., Britton, D., Mitchell, M., Bhimji, W., and Smith, D.
• College/School: College of Science and Engineering > School of Physics and Astronomy
• Journal Name: Journal of Physics: Conference Series
• ISSN: 1742-6588
• Copyright Holders: Copyright © 2013 The Authors
• Publisher Policy: Reproduced under a Creative Commons License

Related URLs

• Organisation