After the training phase, is it better to run neural networks on a GPU or CPU?
This depends on many factors, such as the neural network architecture (CNNs tend to be better optimized than RNN on GPU) as well as how many test samples you give as input to the neural network (GPUs can be even faster when given a batch of samples instead of a single sample).

As an example, here is a benchmark comparing CPU with GPU on different CNN-based architectures. The forward pass is much slower on a CPU in that case:



FYI: Benchmarks based on neural networks libraries to compare the performance between different GPUs