Hameed, H., Usman, M., Tahir, A., Hussain, A., Abbas, H. , Cui, T. J., Imran, M. A. and Abbasi, Q. H. (2022) Pushing the limits of remote RF sensing by reading lips under the face mask. Nature Communications, 13, 5168. (doi: 10.1038/s41467-022-32231-1) (PMID:36071056) (PMCID:PMC9452506)
![[img]](https://eprints.gla.ac.uk/style/images/fileicons/text.png) |
Text
275725.pdf - Published Version Available under License Creative Commons Attribution. 2MB |
Publisher's URL: https://www.nature.com/articles/s41467-022-32231-1
Abstract
The problem of Lip-reading has become an important research challenge in recent years. The goal is to recognise speech from lip movements. Most of the Lip-reading technologies developed so far are camera-based, which require video recording of the target. However, these technologies have well-known limitations of occlusion and ambient lighting with serious privacy concerns. Furthermore, vision-based technologies are not useful for multi-modal hearing aids in the coronavirus (COVID-19) environment, where face masks have become a norm. This paper aims to solve the fundamental limitations of camera-based systems by proposing a radio frequency (RF) based Lip-reading framework, having an ability to read lips under face masks. The framework employs Wi-Fi and radar technologies as enablers of RF sensing based Lip-reading. A dataset comprising of vowels A, E, I, O, U and empty (static/closed lips) is collected using both technologies, with a face mask. The collected data is used to train machine learning (ML) and deep learning (DL) models. A high classification accuracy of 95% is achieved on the Wi-Fi data utilising neural network (NN) models. Moreover, similar accuracy is achieved by VGG16 deep learning model on the collected radar-based dataset.
| Item Type: | Articles |
|---|---|
| Additional Information: | This work was supported in parts by Engineering and Physical Sciences Research Council (EPSRC) grants: EP/T021063/1 (Q.H., M.I, A.H.) and EP/T021020/1 (M.I.). |
| Status: | Published |
| Refereed: | Yes |
| Glasgow Author(s) Enlighten ID: | Tahir, Dr Ahsen and Hameed, Mrs Hira and Imran, Professor Muhammad and Usman, Dr Muhammad and Abbas, Dr Hasan and Abbasi, Dr Qammer |
| Creator Roles: | Hameed, H.Conceptualization, Methodology, Validation, Software, Data curation, Writing – original draft, Writing – review and editing, Visualization Usman, M.Conceptualization, Methodology, Validation, Formal analysis, Software, Data curation, Writing – original draft, Writing – review and editing, Visualization Tahir, A.Methodology, Software, Writing – original draft, Writing – review and editing Abbas, H.Methodology, Writing – review and editing Imran, M. A.Validation, Resources, Writing – review and editing, Supervision, Project administration, Funding acquisition Abbasi, Q. H.Conceptualization, Methodology, Validation, Formal analysis, Resources, Writing – review and editing, Supervision, Project administration, Funding acquisition |
| Authors: | Hameed, H., Usman, M., Tahir, A., Hussain, A., Abbas, H., Cui, T. J., Imran, M. A., and Abbasi, Q. H. |
| College/School: | College of Science and Engineering > School of Engineering College of Science and Engineering > School of Engineering > Autonomous Systems and Connectivity College of Science and Engineering > School of Engineering > Electronics and Nanoscale Engineering |
| Journal Name: | Nature Communications |
| Publisher: | Nature Research |
| ISSN: | 2041-1723 |
| ISSN (Online): | 2041-1723 |
| Copyright Holders: | Copyright © 2022 The Authors |
| First Published: | First published in Nature Communications 13: 5168 |
| Publisher Policy: | Reproduced under a Creative Commons License |
| Related URLs: | |
| Data DOI: | 10.5525/gla.researchdata.1282 |
University Staff: Request a correction | Enlighten Editors: Update this record
Funder and Project Information
Funder and Project Information