

94.4 ms
   


The latency of this model on CPU with all of the tried settings in terms of batch size, concurrency and number of workers did not meet the SLA, in fact ~13x higher.
Moving the model serving to GPU, immediately could improve the latency ~13x from 305 ms down to 23.6 ms. 
One of the simplest optimizations that we could do for the model was lowering its precision to fp16, it is one liner (model.half())  and could reduce the model P99 latency by  32% and increase the throughput by almost the same amount.
There could be other optimization done by Torchscripting the model and using  optimize_for_inference or other tricks including onnx or tensorrt runtime optimizations which leverage aggressive fusions are out of the scope of this post. We will discuss model optimizations in a separate post.