Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition

Men, Q., Ho, E. S.L. , Shum, H. P.H. and Leung, H. (2023) Focalized contrastive view-invariant learning for self-supervised skeleton-based action recognition. Neurocomputing, 537, pp. 198-209. (doi: 10.1016/j.neucom.2023.03.070)

Abstract

Learning view-invariant representation is a key to improving feature discrimination power for skeleton-based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised classifiers with superior recognition performance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.

• Item Type: Articles
• Additional Information: The work described in this paper was supported in part by a grant from City University of Hong Kong (Project No. 9678139) and the Royal Society (Ref: IES\R2\181024 and IES\R1\191147).
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Ho, Dr Edmond S. L
• Authors: Men, Q., Ho, E. S.L., Shum, H. P.H., and Leung, H.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Neurocomputing
• Publisher: Elsevier
• ISSN: 0925-2312
• ISSN (Online): 1872-8286
• Published Online: 31 March 2023
• Copyright Holders: Copyright © 2023 Elsevier B.V.
• First Published: First published in Neurocomputing 537: 198-209
• Publisher Policy: Reproduced in accordance with the publisher copyright policy