We adopt the annotation machinery of BIBREF5 implemented using Amazon's Mechanical Turk, and annotate each predicate by 2 trained workers independently, while a third consolidates their annotations into a final set of roles and arguments.  
 Question: How was the previous dataset annotated?
the annotation machinery of BIBREF5