Humans optional? Automatic large-scale test collections for entity, passage, and entity-passage retrieval

Dietz, L. and Dalton, J. (2020) Humans optional? Automatic large-scale test collections for entity, passage, and entity-passage retrieval. Datenbank-Spektrum, 20, pp. 17-28. (doi: 10.1007/s13222-020-00334-y)

Abstract

Manually creating test collections is a time-, effort-, and cost-intensive process. This paper describes a fully automatic alternative for deriving large-scale test collections, where no human assessments are needed. The empirical experiments confirm that automatic test collection and manual assessments agree on the best performing systems. The collection includes relevance judgments for both text passages and knowledge base entities. Since test collections with relevance data for both entity and text passages are rare, this approach provides a cost-efficient way for training and evaluating ad hoc passage retrieval, entity retrieval, and entity-aware text retrieval methods.

• Item Type: Articles
• Additional Information: This material is based upon work supported by the National Science Foundation under Grant No. 1846017.
• Status: Published
• Refereed: Yes
• Glasgow Author(s) Enlighten ID: Dalton, Dr Jeff
• Authors: Dietz, L., and Dalton, J.
• College/School: College of Science and Engineering > School of Computing Science
• Journal Name: Datenbank-Spektrum
• Publisher: Springer
• ISSN: 1618-2162
• ISSN (Online): 1610-1995
• Published Online: 20 March 2020
• Copyright Holders: Copyright © Gesellschaft für Informatik e.V. and Springer-Verlag GmbH Germany, part of Springer Nature 2020
• First Published: First published in Datenbank-Spektrum 20:17-28
• Publisher Policy: Reproduced in accordance with the publisher copyright policy