A Clarifying Question Selection System from NTES_ALONG in Convai3
  Challenge
  This paper presents the participation of NetEase Game AI Lab team for the
ClariQ challenge at Search-oriented Conversational AI (SCAI) EMNLP workshop in
2020. The challenge asks for a complete conversational information retrieval
system that can understanding and generating clarification questions. We
propose a clarifying question selection system which consists of response
understanding, candidate question recalling and clarifying question ranking. We
fine-tune a RoBERTa model to understand user's responses and use an enhanced
BM25 model to recall the candidate questions. In clarifying question ranking
stage, we reconstruct the training dataset and propose two models based on
ELECTRA. Finally we ensemble the models by summing up their output
probabilities and choose the question with the highest probability as the
clarification question. Experiments show that our ensemble ranking model
outperforms in the document relevance task and achieves the best recall@[20,30]
metrics in question relevance task. And in multi-turn conversation evaluation
in stage2, our system achieve the top score of all document relevance metrics.
