What is the best way to use multiprocessing CPU inference for PyTorch models?
I have to productionize a PyTorch BERT Question Answer model. The CPU inference is very slow for me as for every query the model needs to evaluate 30 samples. Out of the result of these 30 samples, I pick the answer with the maximum score. GPU would be too costly for me to use for inference.

Can I leverage multiprocessing / parallel CPU inference for this?
If Yes, what is the best practice to do so?
If No, is there a cloud option that bills me only for the GPU queries I make and not for continuously running the GPU instance?

Another possible way to get better performance would be to reduce the model as much as possible.

One of the most promising techniques is quantized and binarized neural networks. Here are some references:


https://arxiv.org/abs/1603.05279
https://arxiv.org/abs/1602.02505

