Statistical challenges of high-dimensional data

Johnstone, I.M. and Titterington, D.M. (2009) Statistical challenges of high-dimensional data. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 367(1906), pp. 4237-4253. (doi: 10.1098/rsta.2009.0159)

Abstract

Modern applications of statistical theory and methods can involve extremely large datasets, often with huge numbers of measurements on each of a comparatively small number of experimental units. New methodology and accompanying theory have emerged in response: the goal of this Theme Issue is to illustrate a number of these recent developments. This overview article introduces the difficulties that arise with high-dimensional data in the context of the very familiar linear statistical model: we give a taste of what can nevertheless be achieved when the parameter vector of interest is sparse, that is, contains many zero elements. We describe other ways of identifying low-dimensional subspaces of the data space that contain all useful information. The topic of classification is then reviewed along with the problem of identifying, from within a very large set, the variables that help to classify observations. Brief mention is made of the visualization of high-dimensional data and ways to handle computational problems in Bayesian analysis are described. At appropriate points, reference is made to the other papers in the issue

• Item Type: Articles
• Keywords: Bayesian analysis Classification cluster analysis DANTZIG SELECTOR EIGENMAPS High-dimensional data LASSO Methods MODEL RARE REDUCTION REGRESSION Sparsity USEFUL FEATURES WEAK
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Titterington, Professor D
• Authors: Johnstone, I.M., and Titterington, D.M.
• College/School: College of Science and Engineering > School of Mathematics and Statistics > Statistics
• Journal Name: Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences
• Publisher: Royal Society
• ISSN: 1364-503X