What size of neural networks can be trained on current consumer grade GPUs? (1060,1070,1080) is it possible to give a rule of thumb estimate about the size of neural networks that are trainable on common consumer grade GPUs?
For example:
The Emergence of Locomotion (Reinforcement) paper trains a network using tanh activation of the neurons. They have a 3 layer NN with 300,200,100 units for the Planar Walker. But they don’t report the hardware and time ...
But could a rule of thumb be developed? Also just based on current empirical results, so for example:
$X$ Units using sigmoid activation can run $Y$ learning iterations per hour on a 1060.
Or using activation function $a$ instead of $b$ causes a $n$ times decrease in performance.
If a student/researcher/curious mind is going to buy a GPU for playing around with these networks, how do you decide what you get? A 1060 is apparently the entry level budget option, but how can you evaluate if it is not smarter to just get a crappy netbook instead of building a high power desktop and spend the saved $ on on-demand cloud infrastructure.
Motivation for the question: I just purchased a 1060 and (clever, to ask the question afterwards huh) wonder if I should have just kept the $ and made a Google Cloud account. And if I can run my master thesis simulation on the GPU.
Usually the problem is to fit the model into video RAM.
If it does not, you cannot train your model at all without big efforts (like training parts of the model separately).
If it does, time is your only problem. But the difference in training time between consumer GPUs like the Nvidia 1080 and much more expensive GPU accelerators like the Nvidia K80 are not very large. Actually the best consumer cards are faster than GPU accelerators, but lack other properties like VRAM. Random comparisons and benchmarks: Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning, A Comparison between NVIDIA’s GeForce GTX 1080 and Tesla P100 for Deep Learning.
To calculate if your models fits into VRAM, you just approximate how much data and which hyperparameters you have (inputs, outputs, weights, layers, batch size, which datatype and so on).