What GPU specifications matter when training and using neural networks? I need to purchase some GPUs, which I plan to use for training and using some neural networks (most likely with Theano and Torch). Which GPU specifications should I pay attention to?

E.g.:


one should make sure that the VRAM is large enough for one's application
the more teraflops, the faster programs running exclusively on the GPU will run
the higher the thermal design power (TDP), the higher the electricity bill
the compute capability, which determines which version of CUDA and cuDNN one can use. 


What else matter, and what else doesn't matter?

E.g., is it reasonable to assume that the number of sockets, clock speed and the number of cores do not bring any additional useful information (since we already consider the number of Tflops)?
Besides the criteria listed look for


The number of cores
The ability to do low precision arithmetic


In practice you are probably best off just getting the latest generation NVIDIA card (currently 10xx).