Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions

Hepburn, A. J. and McCreadie, R. (2022) Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions. In: 44th European Conference on Information Retrieval (ECIR 2022), Stavanger, Norway, 10-14 Apr 2022, pp. 137-143. ISBN 9783030997380 (doi: 10.1007/978-3-030-99739-7_16)

Abstract

Transfer learning approaches have shown to significantly improve performance on downstream tasks. However, it is common for prior works to only report where transfer learning was beneficial, ignoring the significant trial-and-error required to find effective settings for transfer. Indeed, not all task combinations lead to performance benefits, and brute-force searching rapidly becomes computationally infeasible. Hence the question arises, can we predict whether transfer between two tasks will be beneficial without actually performing the experiment? In this paper, we leverage explainability techniques to effectively predict whether task pairs will be complementary, through comparison of neural network activation between single-task models. In this way, we can avoid grid-searches over all task and hyperparameter combinations, dramatically reducing the time needed to find effective task pairs. Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.

• Item Type: Conference Proceedings
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Hepburn, Mr Alexander and Mccreadie, Dr Richard
• Authors: Hepburn, A. J., and McCreadie, R.
• Subjects: Q Science > Q Science (General)
• College/School: College of Science and Engineering > School of Computing Science
• Research Centre: College of Science and Engineering > School of Computing Science > IDA Section > GPU Cluster
• Research Group: Information Retrieval
• Publisher: Springer International Publishing
• ISSN: 0302-9743
• ISBN: 9783030997380
• Published Online: 05 April 2022
• Copyright Holders: Copyright © 2022 The Authors
• First Published: First published in Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part II: 137-143
• Publisher Policy: Reproduced in accordance with the publisher copyright policy

Related URLs

• Pre-print
• Organisation