Bi-LSTM network for multimodal continuous human activity recognition and fall detection

Li, H., Shrestha, A., Heidari, H. , Le Kernec, J. and Fioranelli, F. (2020) Bi-LSTM network for multimodal continuous human activity recognition and fall detection. IEEE Sensors Journal, 20(3), pp. 1191-1201. (doi: 10.1109/JSEN.2019.2946095)

199184.pdf - Accepted Version



This paper presents a framework based on multi-layer bi-LSTM network (bidirectional Long Short-Term Memory) for multimodal sensor fusion to sense and classify daily activities’ patterns and high-risk events such as falls. The data collected in this work are continuous activity streams from FMCW radar and three wearable inertial sensors on the wrist, waist, and ankle. Each activity has a variable duration in the data stream so that the transitions between activities can happen at random times within the stream, without resorting to conventional fixed-duration snapshots. The proposed bi-LSTM implements soft feature fusion between wearable sensors and radar data, as well as two robust hard-fusion methods using the confusion matrices of both sensors. A novel hybrid fusion scheme is then proposed to combine soft and hard fusion to push the classification performances to approximately 96% accuracy in identifying continuous activities and fall events. These fusion schemes implemented with the proposed bi-LSTM network are compared with conventional sliding window approach, and all are validated with realistic “leaving one participant out” (L1PO) method (i.e. testing subjects unknown to the classifier). The developed hybrid-fusion approach is capable of stabilizing the classification performance among different participants in terms of reducing accuracy variance of up to 18.1% and increasing minimum, worst-case accuracy up to 16.2%.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Fioranelli, Dr Francesco and Heidari, Dr Hadi and Le Kernec, Dr Julien and Shrestha, Mr Aman and Li, Haobo
Authors: Li, H., Shrestha, A., Heidari, H., Le Kernec, J., and Fioranelli, F.
College/School:College of Science and Engineering > School of Engineering > Electronics and Nanoscale Engineering
College of Science and Engineering > School of Engineering > Systems Power and Energy
Journal Name:IEEE Sensors Journal
ISSN (Online):1558-1748
Published Online:07 October 2019
Copyright Holders:Copyright © 2019 Crown Copyright
First Published:First published in IEEE Sensors Journal 20(3):1191-1201
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
301526Intelligent RF Sensing for Fall and Health PredictionFrancesco FioranelliEngineering and Physical Sciences Research Council (EPSRC)EP/R041679/1ENG - Systems Power & Energy