Title: RELink: A Research Framework and Test Collection for Entity-Relationship Retrieval,
Abstract: Improvements of entity-relationship (E-R) search techniques have been
hampered by a lack of test collections, particularly for complex queries
involving multiple entities and relationships. In this paper we describe a
method for generating E-R test queries to support comprehensive E-R search
experiments. Queries and relevance judgments are created from content that
exists in a tabular form where columns represent entity types and the table
structure implies one or more relationships among the entities. Editorial work
involves creating natural language queries based on relationships represented
by the entries in the table. We have publicly released the RELink test
collection comprising 600 queries and relevance judgments obtained from a
sample of Wikipedia List-of-lists-of-lists tables. The latter comprise tuples
of entities that are extracted from columns and labelled by corresponding
entity types and relationships they represent. In order to facilitate research
in complex E-R retrieval, we have created and released as open source the
RELink Framework that includes Apache Lucene indexing and search specifically
tailored to E-R retrieval. RELink includes entity and relationship indexing
based on the ClueWeb-09-B Web collection with FACC1 text span annotations
linked to Wikipedia entities. With ready to use search resources and a
comprehensive test collection, we support community in pursuing E-R research at
scale.