Pushing the limits of remote RF sensing by reading lips under the face mask

Hameed, H., Usman, M., Tahir, A., Hussain, A., Abbas, H. , Cui, T. J., Imran, M. A. and Abbasi, Q. H. (2022) Pushing the limits of remote RF sensing by reading lips under the face mask. Nature Communications, 13, 5168. (doi: 10.1038/s41467-022-32231-1) (PMID:36071056) (PMCID:PMC9452506)

[img] Text
275725.pdf - Published Version
Available under License Creative Commons Attribution.

2MB

Publisher's URL: https://www.nature.com/articles/s41467-022-32231-1

Abstract

The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.

Item Type:Articles
Additional Information:This work was supported in parts by Engineering and Physical Sciences Research Council (EPSRC) grants: EP/T021063/1 (Q.H., M.I, A.H.) and EP/T021020/1 (M.I.).
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Tahir, Dr Ahsen and Hameed, Mrs Hira and Imran, Professor Muhammad and Usman, Dr Muhammad and Abbas, Dr Hasan and Abbasi, Professor Qammer
Creator Roles:
Hameed, H.Conceptualization, Methodology, Validation, Software, Data curation, Writing – original draft, Writing – review and editing, Visualization
Usman, M.Conceptualization, Methodology, Validation, Formal analysis, Software, Data curation, Writing – original draft, Writing – review and editing, Visualization
Tahir, A.Methodology, Software, Writing – original draft, Writing – review and editing
Abbas, H.Methodology, Writing – review and editing
Imran, M. A.Validation, Resources, Writing – review and editing, Supervision, Project administration, Funding acquisition
Abbasi, Q. H.Conceptualization, Methodology, Validation, Formal analysis, Resources, Writing – review and editing, Supervision, Project administration, Funding acquisition
Authors: Hameed, H., Usman, M., Tahir, A., Hussain, A., Abbas, H., Cui, T. J., Imran, M. A., and Abbasi, Q. H.
College/School:College of Science and Engineering > School of Engineering
College of Science and Engineering > School of Engineering > Autonomous Systems and Connectivity
College of Science and Engineering > School of Engineering > Electronics and Nanoscale Engineering
Journal Name:Nature Communications
Publisher:Nature Research
ISSN:2041-1723
ISSN (Online):2041-1723
Copyright Holders:Copyright © 2022 The Authors
First Published:First published in Nature Communications 13: 5168
Publisher Policy:Reproduced under a Creative Commons License
Related URLs:
Data DOI:10.5525/gla.researchdata.1282

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
307826COG-MHEAR: Towards cognitiveky-inspired 5G-IoT enabled, multi-modal Hearing AidsQammer H AbbasiEngineering and Physical Sciences Research Council (EPSRC)EP/T021063/1ENG - Systems Power & Energy
307829Quantum-Inspired Imaging for Remote Monitoring of Health & Disease in Community HealthcareJonathan CooperEngineering and Physical Sciences Research Council (EPSRC)EP/T021020/1ENG - Biomedical Engineering