Structured Fine-Tuning of Contextual Embeddings for Effective Biomedical Retrieval

Ueda, A., Santos, R., Macdonald, C. and Ounis, I. (2021) Structured Fine-Tuning of Contextual Embeddings for Effective Biomedical Retrieval. In: SIGIR 2021: 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, 11-15 July 2021, pp. 2031-2035. (doi: 10.1145/3404835.3463075)

[img] Text
239428.pdf - Accepted Version

647kB

Abstract

Biomedical literature retrieval has greatly benefited from recent advances in neural language modeling. In particular, fine-tuning pretrained contextual language models has shown impressive results in recent biomedical retrieval evaluation campaigns. Nevertheless, current approaches neglect the inherent structure available from biomedical abstracts, which are (often explicitly) organised into semantically coherent sections such as background, methods, results, and conclusions. In this paper, we investigate the suitability of leveraging biomedical abstract sections for fine-tuning pretrained contextual language models at a finer granularity. Our results on two TREC biomedical test collections demonstrate the effectiveness of the proposed structured fine-tuning regime in contrast to a standard fine-tuning that does not leverage structure. Through an ablation study, we show that models fine-tuned on individual sections are able to capture potentially useful word contexts that may be otherwise ignored by structure-agnostic models.

Item Type:Conference Proceedings
Additional Information:This work was partially funded by the authors’ individual grants from CNPq, CAPES, and FAPEMIG.
Status:Published
Refereed:Yes
Glasgow Author(s) Enlighten ID:Macdonald, Professor Craig and Ounis, Professor Iadh and Ueda, Alberto
Authors: Ueda, A., Santos, R., Macdonald, C., and Ounis, I.
College/School:College of Science and Engineering > School of Computing Science
Published Online:11 July 2021
Copyright Holders:Copyright © 2021 Association for Computing Machinery
Publisher Policy:Reproduced in accordance with the publisher copyright policy

University Staff: Request a correction | Enlighten Editors: Update this record