Can we use a pre trained Encoder (BERT, XLM ) with a Decoder (GPT, Transformer-XL) to build a Chatbot instead of Language Translation?
=> I don't have more ideas about BART and T5 Right now. but I had created a chatbot Based on GPT-2 Model-based on Microsoft DialoGPT. Which is fine-tuned on millions of parameters of Reddit. you can fine-tune on your own data using DialoGPT. I had not found a public decoding method. So, We tried to generate diverse responses using simple nucleus sampling at each time step. So, We used greedy nucleus sampling multiple times in parallel to generate multiple candidates responses. We generated 30 such responses for each turn and also used the last 3 turn conversation history as extra context for generator. So, We can generate responses related to the current Context of Conversation.

We used Code reference from here. you need to implement greedy nucleus sampling multiple on this code.

=>Now To Choose the best-matched response from response generated I created a Reranker.I used two sub-component.

=>Component one Counts Cosine similarity between Response and Query using sentence Encoder.

=>In Component Two we cross-entropy error for generating original query from the response.For that I used reverse-generator of DialogueGPT. 

We Combined Score of Both Component as Score=norm(comp1Score)+norm(1-comp2Loss). Using this score you can find best response.

For Tutorial My Favourite Hugging Face they have BERT,GPT, GPT-2, XLNET,etc...
and Second Favourite Facebook ParlAI, 
try Blender Model also.