Encoder-Decoder performance time
Could you tell us a little bit more about those 2 models ?
If your small network is recurrent, the computation graph of the model is not very parallelizable. It could explain the latency.