Goal-Oriented Multi-Modal Interactive Recommendation with Verbal and Non-Verbal Relevance Feedback

Wu, Y., Macdonald, C. and Ounis, I. (2023) Goal-Oriented Multi-Modal Interactive Recommendation with Verbal and Non-Verbal Relevance Feedback. In: 17th ACM Conference on Recommender Systems (RecSys 2023), Singapore, 18-22 Sept 2023, pp. 362-373. ISBN 9798400702419 (doi: 10.1145/3604915.3608775)

Abstract

Interactive recommendation enables users to provide verbal and non-verbal relevance feedback (such as natural-language critiques and likes/dislikes) when viewing a ranked list of recommendations (such as images of fashion products), in order to guide the recommender system towards their desired items (i.e. goals) across multiple interaction turns. Such a multi-modal interactive recommendation (MMIR) task has been successfully formulated with deep reinforcement learning (DRL) algorithms by simulating the interactions between an environment (i.e. a user) and an agent (i.e. a recommender system). However, it is typically challenging and unstable to optimise the agent to improve the recommendation quality associated with implicit learning of multi-modal representations in an end-to-end fashion in DRL. This is known as the coupling of policy optimisation and representation learning. To address this coupling issue, we propose a novel goal-oriented multi-modal interactive recommendation model (GOMMIR) that uses both verbal and non-verbal relevance feedback to effectively incorporate the users’ preferences over time. Specifically, our GOMMIR model employs a multi-task learning approach to explicitly learn the multi-modal representations using a multi-modal composition network when optimising the recommendation agent. Moreover, we formulate the MMIR task using goal-oriented reinforcement learning and enhance the optimisation objective by leveraging non-verbal relevance feedback for hard negative sampling and providing extra goal-oriented rewards to effectively optimise the recommendation agent. Following previous work, we train and evaluate our GOMMIR model by using user simulators that can generate natural-language feedback about the recommendations as a surrogate for real human users. Experiments conducted on four well-known fashion datasets demonstrate that our proposed GOMMIR model yields significant improvements in comparison to the existing state-of-the-art baseline models.

• Item Type: Conference Proceedings
• Additional Information: The authors acknowledge support from EPSRC grant EP/R018634/1 entitled Closed-Loop Data Science for Complex, Computationally-and Data-Intensive Analytics.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Wu, Mr Yaxiong and Ounis, Professor Iadh and Macdonald, Professor Craig
• Authors: Wu, Y., Macdonald, C., and Ounis, I.
• College/School: College of Science and Engineering; College of Science and Engineering > School of Computing Science
• Research Centre: College of Science and Engineering > School of Computing Science > IDA Section > GPU Cluster
• ISBN: 9798400702419
• Copyright Holders: © 2023 Copyright held by the owner/author(s).
• First Published: First published in RecSys '23: Proceedings of the 17th ACM Conference on Recommender Systems
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Organisation

Project Code	Award No	Project Name	Principal Investigator	Funder's Name	Funder Ref	Lead Dept
300982		Exploiting Closed-Loop Aspects in Computationally and Data Intensive Analytics	Roderick Murray-Smith	Engineering and Physical Sciences Research Council (EPSRC)	EP/R018634/1	Computing Science