How do GPUs faciliate the training of a Deep Learning Architecture?
GPUs are able to execute a huge amount of similar and simple instructions (floating point operations like addition and multiplication) in parallel. In contrast to a CPU which is able to execute a few complex tasks sequentially very quick. Therefore GPUs are very good at doing vector &amp; matrix operations.

If you look at the operations performed inside a single basic NN layer you will see that most of the operations are matrix-vector multiplications:

$$x_{i+1} = \sigma(W_ix_i + b_i)$$

where $x_i$ is the input vector, $W_i$ the matrix of weights, $b_i$ the bias vector in the $i^{th}$ layer, $x_{i+1}$ the output vector and $\sigma(\cdot)$ the elemtwise non-linear activation. The computational complexity here is governed by the matrix-vector multiplication. If you look at the architecture of a LSTM cell you will notice that inside of it are multiple such operations.

Being able to execute the matrix-vector operations quickly and efficiently in parallel will reduce execution time, this is where GPUs excel CPUs.