Q-learning: flexible learning about useful utilities

Moodie, E. E.M., Dean, N. and Sun, Y. R. (2014) Q-learning: flexible learning about useful utilities. Statistics in Biosciences, 6(2), pp. 223-243. (doi: 10.1007/s12561-013-9103-z)

Abstract

Dynamic treatment regimes are fast becoming an important part of medicine, with the corresponding change in emphasis from treatment of the disease to treatment of the individual patient. Because of the limited number of trials to evaluate personally tailored treatment sequences, inferring optimal treatment regimes from observational data has increased importance. Q-learning is a popular method for estimating the optimal treatment regime, originally in randomized trials but more recently also in observational data. Previous applications of Q-learning have largely been restricted to continuous utility end-points with linear relationships. This paper is the first attempt at both extending the framework to discrete utilities and implementing the modelling of covariates from linear to more flexible modelling using the generalized additive model (GAM) framework. Simulated data results show that the GAM adapted Q-learning typically outperforms Q-learning with linear models and other frequently-used methods based on propensity scores in terms of coverage and bias/MSE. This represents a promising step toward a more fully general Q-learning approach to estimating optimal dynamic treatment regimes.

• Item Type: Articles
• Keywords: Dynamic Treatment Regimes, Q-learning, Generalized Additive Models, Discrete data
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Dean, Dr Nema
• Authors: Moodie, E. E.M., Dean, N., and Sun, Y. R.
• Subjects: H Social Sciences > HA Statistics
• College/School: College of Science and Engineering > School of Mathematics and Statistics > Statistics
• Research Group: Statistical Methodology
• Journal Name: Statistics in Biosciences
• Journal Abbr.: Stat Biosci
• Publisher: Springer US
• ISSN: 1867-1764
• ISSN (Online): 1867-1772
• Copyright Holders: Copyright © 2014 Springer
• First Published: First published in Statistics in Biosciences 6()2):223-243
• Publisher Policy: Reproduced in accordance with the copyright policy of the publisher.