Detecting egregious responses in neural sequence-to-sequence models
  In this work, we attempt to answer a critical question: whether there exists
some input sequence that will cause a well-trained discrete-space neural
network sequence-to-sequence (seq2seq) model to generate egregious outputs
(aggressive, malicious, attacking, etc.). And if such inputs exist, how to find
them efficiently. We adopt an empirical methodology, in which we first create
lists of egregious output sequences, and then design a discrete optimization
algorithm to find input sequences that will cause the model to generate them.
Moreover, the optimization algorithm is enhanced for large vocabulary search
and constrained to search for input sequences that are likely to be input by
real-world users. In our experiments, we apply this approach to dialogue
response generation models trained on three real-world dialogue data-sets:
Ubuntu, Switchboard and OpenSubtitles, testing whether the model can generate
malicious responses. We demonstrate that given the trigger inputs our algorithm
finds, a significant number of malicious sentences are assigned large
probability by the model, which reveals an undesirable consequence of standard
seq2seq training.
