Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review

Igelström, E. , Campbell, M. , Craig, P. and Katikireddi, S. V. (2021) Cochrane's risk of bias tool for non-randomized studies (ROBINS-I) is frequently misapplied: A methodological systematic review. Journal of Clinical Epidemiology, 140, pp. 22-32. (doi: 10.1016/j.jclinepi.2021.08.022) (PMID:34437948) (PMCID:PMC8809341)

[img] Text
250140.pdf - Published Version
Available under License Creative Commons Attribution.



Objectives: We aimed to review how ‘Risk of Bias In Non-randomized Studies–of Interventions’ (ROBINS-I), a Cochrane risk of bias assessment tool, has been used in recent systematic reviews. Study Design and Setting: Database and citation searches were conducted in March 2020 to identify recently published reviews using ROBINS-I. Reported ROBINS-I assessments and data on how ROBINS-I was used were extracted from each review. Methodological quality of reviews was assessed using AMSTAR 2 (‘A MeaSurement Tool to Assess systematic Reviews’). Results: Of 181 hits, 124 reviews were included. Risk of bias was serious/critical in 54% of assessments on average, most commonly due to confounding. Quality of reviews was mostly low, and modifications and incorrect use of ROBINS-I were common, with 20% reviews modifying the rating scale, 20% understating overall risk of bias, and 19% including critical-risk of bias studies in evidence synthesis. Poorly conducted reviews were more likely to report low/moderate risk of bias (predicted probability 57% [95% CI: 47–67] in critically low-quality reviews, 31% [19–46] in high/moderate-quality reviews). Conclusion: Low-quality reviews frequently apply ROBINS-I incorrectly, and may thus inappropriately include or give too much weight to uncertain evidence. Readers should be aware that such problems can lead to incorrect conclusions in reviews.

Item Type:Articles
Additional Information:E.I., M.C., P.C., and S.V.K. receive funding from the Medical Research Council (MC_UU_00022/2) and the Scottish Government Chief Scientist Office (SPHSU17). S.V.K. is supported by an NHS Research Scotland Senior Clinical Fellowship (SCAF/15/02).
Glasgow Author(s) Enlighten ID:Katikireddi, Professor Vittal and Craig, Professor Peter and Igelström, Erik and Campbell, Ms Mhairi
Creator Roles:
Igelström, E.Conceptualization, Methodology, Investigation, Formal analysis, Writing – original draft
Campbell, M.Validation, Writing – review and editing
Craig, P.Validation, Writing – review and editing
Katikireddi, S. V.Conceptualization, Supervision, Validation, Writing – review and editing
Authors: Igelström, E., Campbell, M., Craig, P., and Katikireddi, S. V.
College/School:College of Medical Veterinary and Life Sciences > School of Health & Wellbeing > MRC/CSO SPHSU
Journal Name:Journal of Clinical Epidemiology
ISSN (Online):1878-5921
Published Online:23 August 2021
Copyright Holders:Copyright © 2021 The Authors
First Published:First published in Journal of Clinical Epidemiology 140:22-32
Publisher Policy:Reproduced under a Creative Commons License

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
3048230021Inequalities in healthAlastair LeylandMedical Research Council (MRC)MC_UU_00022/2HW - MRC/CSO Social and Public Health Sciences Unit
3048230071Inequalities in healthAlastair LeylandOffice of the Chief Scientific Adviser (CSO)SPHSU17HW - MRC/CSO Social and Public Health Sciences Unit
172690Understanding the impacts of welfare policy on health: A novel data linkage studySrinivasa KatikireddiOffice of the Chief Scientific Adviser (CSO)SCAF/15/02SHW - Public Health