Should I use GPU or CPU for inference? I'm running a deep learning neural network that has been trained by a GPU. I now want to deploy this to multiple hosts for inference. The question is what are the conditions to decide whether I should use GPU's or CPUs for inference?



Adding more details from comments below. 

I'm new to this so guidance is appreciated. 


Memory: GPU is K80
Framework: Cuda and cuDNN 
Data size per workloads: 20G
Computing nodes to consume: one per job, although would like to consider a scale option 
Cost: I can afford a GPU option if the reasons make sense
Deployment: Running on own hosted bare metal servers, not in the cloud.


Right now I'm running on CPU simply because the application runs ok. But outside of that reason, I'm unsure why one would even consider GPU.
It is true that for training a lot of the parallalization can be exploited by the GPU's, resulting in much faster training. For Inference, this parallalization can be way less, however CNN's will still get an advantage from this resulting in faster inference. Now you just have to ask yourself: is faster inference important? Do I want this extra dependencies (a good GPU, the right files installed etc)?
If speed is not an issue, go for CPU. However note that GPU's can make it an order of magnitude faster in my experience.