Improving Negative Sampling for Word Representation Using Self-embedded Features

Chen, L., Yuan, F., Jose, J. M. and Zhang, W. (2018) Improving Negative Sampling for Word Representation Using Self-embedded Features. In: The 11th International Conference on Web Searching and Data Mining (WSDM 2018), Los Angeles, CA, USA, 05-09 Feb 2018, pp. 99-107. ISBN 9781450355810 (doi: 10.1145/3159652.3159695)

153128.pdf - Accepted Version



Although the word-popularity based negative sampler has shown superb performance in the skip-gram model, the theoretical motivation behind oversampling popular (non-observed) words as negative samples is still not well understood. In this paper, we start from an investigation of the gradient vanishing issue in the skip-gram model without a proper negative sampler. By performing an insightful analysis from the stochastic gradient descent (SGD) learning perspective, we demonstrate, both theoretically and intuitively, negative samples with larger inner product scores are more informative than those with lower scores for the SGD learner in terms of both convergence rate and accuracy. Understanding this, we propose an alternative sampling algorithm that dynamically selects informative negative samples during each SGD update. More importantly, the proposed sampler accounts for multi-dimensional self-embedded features during the sampling process, which essentially makes it more effective than the original popularity-based (one-dimensional) sampler. Empirical experiments further verify our observations and show that our fine-grained samplers gain significant improvement over the existing ones without increasing computational complexity.

Item Type:Conference Proceedings
Additional Information:This work was also partly supported by NSF grant #61572223.
Glasgow Author(s) Enlighten ID:Jose, Professor Joemon and Chen, Dr Long and YUAN, FAJIE
Authors: Chen, L., Yuan, F., Jose, J. M., and Zhang, W.
College/School:College of Science and Engineering > School of Computing Science
Copyright Holders:Copyright © 2018 The Authors
Publisher Policy:Reproduced in accordance with the copyright policy of the publisher
Related URLs:

University Staff: Request a correction | Enlighten Editors: Update this record

Project CodeAward NoProject NamePrincipal InvestigatorFunder's NameFunder RefLead Dept
643481A Situation-aware information infrastructureDimitrios PezarosEngineering and Physical Sciences Research Council (EPSRC)EP/L026015/1COM - COMPUTING SCIENCE