Rating norms should be calculated from cumulative link mixed effects models

Taylor, J. E. , Rousselet, G. A. , Scheepers, C. and Sereno, S. C. (2023) Rating norms should be calculated from cumulative link mixed effects models. Behavior Research Methods, 55, pp. 2175-2196. (doi: 10.3758/s13428-022-01814-7) (PMID:36103049) (PMCID:PMC10439063)

Abstract

Studies which provide norms of Likert ratings typically report per-item summary statistics. Traditionally, these summary statistics comprise the mean and the standard deviation (SD) of the ratings, and the number of observations. Such summary statistics can preserve the rank order of items, but provide distorted estimates of the relative distances between items because of the ordinal nature of Likert ratings. Inter-item relations in such ordinal scales can be more appropriately modelled by cumulative link mixed effects models (CLMMs). In a series of simulations, and with a reanalysis of an existing rating norms dataset, we show that CLMMs can be used to more accurately norm items, and can provide summary statistics analogous to the traditionally reported means and SDs, but which are disentangled from participants’ response biases. CLMMs can be applied to solve important statistical issues that exist for more traditional analyses of rating norms.

• Item Type: Articles
• Additional Information: Funding: This research was supported in part by an Economic and Social Research Council (ESRC) postgraduate fellowship awarded to J. E. Taylor (reference: ES/P000681/l).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Sereno, Dr Sara and Scheepers, Dr Christoph and Rousselet, Dr Guillaume and Taylor, Jack
• Authors: Taylor, J. E., Rousselet, G. A., Scheepers, C., and Sereno, S. C.
• College/School: College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
• Journal Name: Behavior Research Methods
• Publisher: Springer for Psychonomic Society
• ISSN: 1554-351X
• ISSN (Online): 1554-3528
• Published Online: 14 September 2022
• Copyright Holders: Copyright © The Author(s) 2022
• First Published: First published in Behavior Research Methods 55: 2175–2196
• Publisher Policy: Reproduced under a Creative Commons License