Spatio-temporal manifold learning for human motions via long-horizon modeling

Wang, H., Ho, E. S. L. , Shum, H. P. H. and Zhu, Z. (2021) Spatio-temporal manifold learning for human motions via long-horizon modeling. IEEE Transactions on Visualization and Computer Graphics, 27(1), pp. 216-227. (doi: 10.1109/TVCG.2019.2936810) (PMID:31443030)

Full text not currently available from Enlighten.


—Data-driven modeling of human motions is ubiquitous in computer graphics and computer vision applications, such as synthesizing realistic motions or recognizing actions. Recent research has shown that such problems can be approached by learning a natural motion manifold using deep learning on a large amount data, to address the shortcomings of traditional data-driven approaches. However, previous deep learning methods can be sub-optimal for two reasons. First, the skeletal information has not been fully utilized for feature extraction. Unlike images, it is difficult to define spatial proximity in skeletal motions in the way that deep networks can be applied for feature extraction. Second, motion is time-series data with strong multi-modal temporal correlations between frames. On the one hand, a frame could be followed by several candidate frames leading to different motions; on the other hand, long-range dependencies exist where a number of frames in the beginning are correlated with a number of frames later. Ineffective temporal modeling would either under-estimate the multi-modality and variance, resulting in featureless mean motion or over-estimate them resulting in jittery motions, which is a major source of visual artifacts. In this paper, we propose a new deep network to tackle these challenges by creating a natural motion manifold that is versatile for many applications. The network has a new spatial component for feature extraction. It is also equipped with a new batch prediction model that predicts a large number of frames at once, such that longterm temporally-based objective functions can be employed to correctly learn the motion multi-modality and variances. With our system, long-duration motions can be predicted/synthesized using an open-loop setup where the motion retains the dynamics accurately. It can also be used for denoising corrupted motions and synthesizing new motions with given control signals. We demonstrate that our system can create superior results comparing to existing work in multiple applications.

Item Type:Articles
Glasgow Author(s) Enlighten ID:Ho, Dr Edmond S. L
Authors: Wang, H., Ho, E. S. L., Shum, H. P. H., and Zhu, Z.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:IEEE Transactions on Visualization and Computer Graphics
ISSN (Online):1941-0506
Published Online:22 August 2019

University Staff: Request a correction | Enlighten Editors: Update this record