The semi-parametric models were trained on 32 GPUs with each replica split over 2 GPUs, one to train the translation model and the other for computing the CSTM. 
 Question: Does their combination of a non-parametric retrieval and neural network get trained end-to-end?
Yes