 The selection model selects the best answer from the set $\lbrace a_i\rbrace _{i=1}^N$ observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants. We train another neural network to pick the best answer from the candidates. We frame the task as binary classification, distinguishing between above and below average performance. In training, we compute the F1 score of the answer for every instance. If the rewrite produces an answer with an F1 score greater than the average score of the other rewrites the instance is assigned a positive label. 
 Question: how are multiple answers from multiple reformulated questions aggregated?
The selection model selects the best answer from the set $\lbrace a_i\rbrace _{i=1}^N$ observed during the interaction by predicting the difference of the F1 score to the average F1 of all variants.