Applying the ROBINS-I tool to natural experiments: an example from public health

Thomson, H. , Craig, P. , Hilton-Boon, M. , Campbell, M. and Katikireddi, S. V. (2018) Applying the ROBINS-I tool to natural experiments: an example from public health. Systematic Reviews, 7, 15. (doi: 10.1186/s13643-017-0659-4) (PMID:29368630) (PMCID:PMC5784724)

[img]
Preview
Text
148062.pdf - Published Version
Available under License Creative Commons Attribution.

496kB

Abstract

Background: A new tool to assess Risk of Bias In Non-randomised Studies of Interventions (ROBINS-I) was published in Autumn 2016. ROBINS-I uses the Cochrane-approved risk of bias (RoB) approach and focusses on internal validity. As such, ROBINS-I represents an important development for those conducting systematic reviews which include non-randomised studies (NRS), including public health researchers. We aimed to establish the applicability of ROBINS-I using a group of NRS which have evaluated non-clinical public health natural experiments. Methods: Five researchers, all experienced in critical appraisal of non-randomised studies, used ROBINS-I to independently assess risk of bias in five studies which had assessed the health impacts of a domestic energy efficiency intervention. ROBINS-I assessments for each study were entered into a database and checked for consensus across the group. Group discussions were used to identify reasons underpinning lack of consensus for specific questions and bias domains. Results: ROBINS-I helped to systematically articulate sources of bias in NRS. However, the lack of consensus in assessments for all seven bias domains raised questions about ROBINS-I’s reliability and applicability for natural experiment studies. The two RoB domains with least consensus were selection (Domain 2) and performance (Domain 4). Underlying the lack of consensus were difficulties in applying an intention to treat or per protocol effect of interest to the studies. This was linked to difficulties in determining whether the intervention status was classified retrospectively at follow-up, i.e. post hoc. The overall risk of bias ranged from moderate to critical; this was most closely linked to the assessment of confounders. Conclusion: The ROBINS-I tool is a conceptually rigorous tool which focusses on risk of bias due to the counterfactual. Difficulties in applying ROBINS-I may be due to poor design and reporting of evaluations of natural experiments. While the quality of reporting may improve in the future, improved guidance on applying ROBINS-I is needed to enable existing evidence from natural experiments to be assessed appropriately and consistently. We hope future refinements to ROBINS-I will address some of the issues raised here to allow wider use of the tool.

Item Type:Articles
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Katikireddi, Professor Vittal and Craig, Professor Peter and Thomson, Dr Hilary and Campbell, Ms Mhairi and Hilton Boon, Dr Michele
Authors: Thomson, H., Craig, P., Hilton-Boon, M., Campbell, M., and Katikireddi, S. V.
College/School:College of Medical Veterinary and Life Sciences > School of Health & Wellbeing > MRC/CSO SPHSU
Journal Name:Systematic Reviews
Publisher:BioMed Central
ISSN:2046-4053
ISSN (Online):2046-4053
Copyright Holders:Copyright © 2018 The Authors
First Published:First published in Systematic Reviews 7:15
Publisher Policy:Reproduced under a Creative Commons License

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
727651Measuring and Analysing Socioeconomic Inequalities in HealthAlastair LeylandMedical Research Council (MRC)MC_UU_12017/13HW - MRC/CSO Social and Public Health Sciences Unit
727671Informing Healthy Public PolicyPeter CraigMedical Research Council (MRC)MC_UU_12017/15HW - MRC/CSO Social and Public Health Sciences Unit
699162Understanding the impacts of welfare policy on health: A novel data linkage studySrinivasa KatikireddiOffice of the Chief Scientist (CSO)SCAF/15/02IHW - MRC/CSO SPHU