ABNIRML: Analyzing the behavior of neural IR models

MacAvaney, S. , Feldman, S., Goharian, N., Downey, D. and Cohan, A. (2022) ABNIRML: Analyzing the behavior of neural IR models. Transactions of the Association for Computational Linguistics, 10, pp. 224-239. (doi: 10.1162/tacl_a_00457)

[img] Text
259712.pdf - Published Version
Available under License Creative Commons Attribution.



Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search. However, it is not yet well understood why these methods are so effective, what makes some variants more effective than others, and what pitfalls they may have. We present a new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic probes that allow us to test several characteristics—such as writing styles, factuality, sensitivity to paraphrasing and word order—that are not addressed by previous techniques. To demonstrate the value of the framework, we conduct an extensive empirical study that yields insights into the factors that contribute to the neural model’s gains, and identify potential unintended biases the models exhibit. Some of our results confirm conventional wisdom, for example, that recent neural ranking models rely less on exact term overlap with the query, and instead leverage richer linguistic information, evidenced by their higher sensitivity to word and sentence order. Other results are more surprising, such as that some models (e.g., T5 and ColBERT) are biased towards factually correct (rather than simply relevant) texts. Further, some characteristics vary even for the same base language model, and other characteristics can appear due to random variations during model training.

Item Type:Articles
Glasgow Author(s) Enlighten ID:MacAvaney, Dr Sean
Authors: MacAvaney, S., Feldman, S., Goharian, N., Downey, D., and Cohan, A.
College/School:College of Science and Engineering > School of Computing Science
Journal Name:Transactions of the Association for Computational Linguistics
Publisher:MIT Press
Published Online:18 March 2022
Copyright Holders:Copyright © 2022 Association for Computational Linguistics
First Published:First published in Transactions of the Association for Computational Linguistics 10: 224-239
Publisher Policy:Reproduced under a Creative Commons License

University Staff: Request a correction | Enlighten Editors: Update this record