Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition
Top Cited Papers
Open Access
- 18 January 2016
- Vol. 16 (1), 115
- https://doi.org/10.3390/s16010115
Abstract
Human activity recognition (HAR) tasks have traditionally been solved using engineered features obtained by heuristic processes. Current research suggests that deep convolutional neural networks are suited to automate feature extraction from raw sensor inputs. However, human activities are made of complex sequences of motor movements, and capturing this temporal dynamics is fundamental for successful HAR. Based on the recent success of recurrent neural networks for time series domains, we propose a generic deep framework for activity recognition based on convolutional and LSTM recurrent units, which: (i) is suitable for multimodal wearable sensors; (ii) can perform sensor fusion naturally; (iii) does not require expert knowledge in designing features; and (iv) explicitly models the temporal dynamics of feature activations. We evaluate our framework on two datasets, one of which has been used in a public activity recognition challenge. Our results show that our framework outperforms competing deep non-recurrent networks on the challenge dataset by 4% on average; outperforming some of the previous reported results by up to 9%. Our results show that the framework can be applied to homogeneous sensor modalities, but can also fuse multimodal sensors to improve performance. We characterise key architectural hyperparameters’ influence on performance to provide insights about their optimisation.Keywords
Funding Information
- Google Faculty Research Award (Is deep learning useful for wearable activity recognition?)
- EPSRC (EP/N007816/1)
This publication has 31 references indexed in Scilit:
- In-Home Activity Recognition: Bayesian Inference for Hidden Markov ModelsIEEE Pervasive Computing, 2014
- A tutorial on human activity recognition using body-worn inertial sensorsACM Computing Surveys, 2014
- The Opportunity challenge: A benchmark database for on-body sensor-based activity recognitionPattern Recognition Letters, 2013
- The mobile fitness coach: Towards individualized skill assessment using personalized mobile devicesPervasive and Mobile Computing, 2013
- A review of wearable sensors and systems with application in rehabilitationJournal of NeuroEngineering and Rehabilitation, 2012
- Preprocessing techniques for context recognition from accelerometer dataPersonal and Ubiquitous Computing, 2010
- Keeping the Resident in the Loop: Adapting the Smart Home to the UserIEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans, 2009
- Activity identification using body-mounted sensors—a review of classification techniquesPhysiological Measurement, 2009
- Wearable Activity Tracking in Car ManufacturingIEEE Pervasive Computing, 2008
- Turing computability with neural netsApplied Mathematics Letters, 1991