Choosing between CPU and GPU for training a neural network
Unlike some of the other answers, I would highly advice against always training on GPUs without any second thought. This is driven by the usage of deep learning methods on images and texts, where the data is very rich (e.g. a lot of pixels = a lot of variables) and the model similarly has many millions of parameters. For other domains, this might not be the case.


  What is meant by 'small'? For example, would a single-layer MLP with 100 hidden units be 'small'?


Yes, that is definitely very small by modern standards. Unless you have a GPU suited perfectly for training (e.g. NVIDIA 1080 or NVIDIA Titan), I wouldn't be surprised to find that your CPU was faster.

Note that the complexity of your neural network also depends on your number of input features, not just the number of units in your hidden layer. If your hidden layer has 100 units and each observation in your dataset has 4 input features, then your network is tiny (~400 parameters). If each observation instead has 1M input features as in some medical/biotech contexts, then your network is pretty big in terms of number of parameters. For the remainder of my answer I'm assuming you have quite few input features pr. observation.

One good example I've found of comparing CPU vs. GPU performance was when I trained a poker bot using reinforcement learning. For reinforcement learning you often don't want that many layers in your neural network and we found that we only needed a few layers with few parameters. Moreover, the number of input features was quite low. Initially I trained on a GPU (NVIDIA Titan), but it was taking a long time as reinforcement learning requires a lot of iterations. Luckily, I found that training on my CPU instead made my training go 10x as fast! This is just to say that CPU's can sometimes be better for training.


  Are there any other criteria that should be considered when deciding whether to train on CPU or GPU?


It's important to note that while on a GPU you will always want to fill up the entire GPU memory by increasing your batch size, that is not the case on the CPU. On the CPU an increase in batch size will increase the time pr. batch. Therefore, if it's important for you to have a very large batch size (e.g. due to a very noisy signal), it can be beneficial to use a GPU. I haven't experienced this in practice though and normally small batch sizes are preferred.