Extracting visual micro-doppler signatures from human lips motion using UoG radar sensing data for hearing aid applications

Saeed, U., Shah, S. A., Ghadi, Y. Y., Khan, M. Z., Ahmad, J., Shah, S. I., Hameed, H. and Abbasi, Q. (2023) Extracting visual micro-doppler signatures from human lips motion using UoG radar sensing data for hearing aid applications. IEEE Sensors Journal, 23(19), pp. 22111-22118. (doi: 10.1109/JSEN.2023.3308972)

Abstract

This study proposes a secure and effective lips-reading system that can accurately detect lips movements, even when face masks are worn. The system utilizes radio frequency (RF) sensing and ultra-wideband (UWB) radar technology, which overcomes the challenges posed by traditional vision-based systems. By leveraging deep learning models, the system interprets lips and mouth movements and achieves an overall accuracy of 90% for both mask-on and mask-off scenarios. The study utilized a trusted dataset from the University of Glasgow (UoG), consisting of spectrograms of lips motions stating five vowels and a voiceless class from distinct participants. The cutting-edge deep learning algorithm, residual neural network (ResNet50), was used for the evaluation of the dataset and achieved an 87% accurate detection rate with a mask-on scenario, which is a 14% improvement compared to prior published work. The findings of this study contribute to the development of a robust lips-reading framework that can enhance communication accessibility in applications such as hearing aids, voice-controlled systems, biometrics, and more.

• Item Type: Articles
• Additional Information: This work is supported in parts by EPSRC grant (EP/W037076/1).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Khan, Muhammad Zakir and Hameed, Mrs Hira and Abbasi, Professor Qammer
• Authors: Saeed, U., Shah, S. A., Ghadi, Y. Y., Khan, M. Z., Ahmad, J., Shah, S. I., Hameed, H., and Abbasi, Q.
• College/School: College of Science and Engineering > School of Engineering > Electronics and Nanoscale Engineering
• Journal Name: IEEE Sensors Journal
• Publisher: IEEE
• ISSN: 1530-437X
• ISSN (Online): 1558-1748
• Copyright Holders: Copyright © 2023 IEEE
• First Published: First published in IEEE Sensors Journal 23(19):22111-22118
• Publisher Policy: Reproduced in accordance with the publisher copyright policy