Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker

Haider, C. L., Suess, N., Hauswald, A., Park, H. and Weisz, N. (2022) Masking of the mouth area impairs reconstruction of acoustic speech features and higher-level segmentational features in the presence of a distractor speaker. NeuroImage, 252, 119044. (doi: 10.1016/j.neuroimage.2022.119044) (PMID:35240298)

Abstract

Multisensory integration enables stimulus representation even when the sensory input in a single modality is weak. In the context of speech, when confronted with a degraded acoustic signal, congruent visual inputs promote comprehension. When this input is masked, speech comprehension consequently becomes more difficult. But it still remains inconclusive which levels of speech processing are affected under which circumstances by occluding the mouth area. To answer this question, we conducted an audiovisual (AV) multi-speaker experiment using naturalistic speech. In half of the trials, the target speaker wore a (surgical) face mask, while we measured the brain activity of normal hearing participants via magnetoencephalography (MEG). We additionally added a distractor speaker in half of the trials in order to create an ecologically difficult listening situation. A decoding model on the clear AV speech was trained and used to reconstruct crucial speech features in each condition. We found significant main effects of face masks on the reconstruction of acoustic features, such as the speech envelope and spectral speech features (i.e. pitch and formant frequencies), while reconstruction of higher level features of speech segmentation (phoneme and word onsets) were especially impaired through masks in difficult listening situations. As we used surgical face masks in our study, which only show mild effects on speech acoustics, we interpret our findings as the result of the missing visual input. Our findings extend previous behavioural results, by demonstrating the complex contextual effects of occluding relevant visual information on speech processing.

• Item Type: Articles
• Additional Information: This work is supported by the Austrian Science Fund, P31230 (“Audiovisual speech entrainment in deafness”) and P34237 (“Impact of face masks on speech comprehension”). Sound icon made by Smashicon from www.flaticon.com.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Park, Dr Hyojin
• Authors: Haider, C. L., Suess, N., Hauswald, A., Park, H., and Weisz, N.
• College/School: College of Medical Veterinary and Life Sciences > School of Psychology & Neuroscience
• Journal Name: NeuroImage
• Publisher: Elsevier
• ISSN: 1053-8119
• ISSN (Online): 1095-9572
• Published Online: 28 February 2022
• Copyright Holders: Copyright © 2022 The Authors
• First Published: First published in NeuroImage 252: 119044
• Publisher Policy: Reproduced under a Creative Commons License