Predict output sequence one at a time with feedback I would like to solve the following classification problem: Given an input sequence and zero or more initial terms of the true output, predict the next term in the output sequence.

For example, my training set is a large corpus of question and answer pairs. Now I would like to predict the response to "How are you?" one word at a time.

Some reasonable responses might be "I'm fine.", "OK.", "Very well, thanks." etc. So the predicted distribution over possible first words would include the first words in those examples, among others.

But now if I see the first word is "Very", I would like my prediction for the second word to reflect that "OK" is now less likely, because "Very OK" is not a common response.

And I would like to repeat this process of predicting the next word given the input sentence and every word so far in the output sentence.

One approach might be to train a stateful RNN on examples like How are you&lt;END&gt; I am fine&lt;END&gt;. But if I understand correctly this would also learn to predict words in the "How are you" part, which I think might detract from the goal.

Maybe separate layers for the input and partial output that are then merged into a decoder layer?
I assume, that your data looks somehow like this (a pair of question and answer)

Question,Answer
How are you?,I'm fine.
How are you?,OK.
How are you?,Very well, thanks.


I'd suggest to transform it in three features: Question  (remains the same), 
InputSeq is the prefix of the answer generated in all possible lengths between zero and the number of words minus one.

Finally the NextWord is the expected result.

So for your initial data you ends with the following setup.

Question, InputSeq, NextWord
How are you,,I'm
How are you,I'm,fine
How are you,,OK
How are you,,Very
How are you,Very,well
How are you,Very well,thanks


The model takes Question and InputSeq   as features and predicts the NextWord.