How does GPU utilization work in the context of neural network training?
I am using an AWS p3.2xlarge instance with the Deep Learning AMI (DLAMI). This instance has a single Tesla V100 (640 Tensor Cores and 5,120 CUDA Cores). When I run the PyTorch Seq2Seq Jupyter Notebook, I noticed that only 25% of the GPU is used. I monitor the GPU usage with the following command watch -n 1 nvidia-smi. 

My question is, what determines GPU usage? Or, why is the GPU usage not 100%? The reason behind this question is related not only to inefficiency that may be a result of code but also cost ($3.06/hour). I am wondering if there is anything more that I can do to maximize the GPU usage.

Of course, this is a deep learning model that is being learned, and the training code sends one sample at a time through the network for learning. I am thinking that mini-batch learning may not be appropriate (e.g. sending a couple of samples through before backpropagating). I am also wondering if the network architecture (the number of layers, their parameters, their input tensor dimensions, etc.) constrains how GPU is being used. For example, if I add more layers or add more hidden nodes, should I expect GPU usage to go up?

The power of GPUs over CPUs is to run many operations at the same time.  However archiving this high level of parallelization is not always easy. Frameworks like Tensorflow or PyTorch do its best to optimise everything for GPU and parallelisation, but this is not possible for every case.

Computations in LSTMs and RNNs in general can be only parallelized to a very limited degree. The problem lies in their sequential structure, LSTMs and RNNs process only one input at a time, and they need to process everything in chronological order (to compute n+1 you always need to compute n before) - otherwise it wouldn't make sense.

So the natural way of processing data in RNNs is completely the opposite of parallelization, using mini-batching does help a lot, but does not solve the fundamental problem of LSTMs.

If you wan't a high amount of parallelization you need to use architectures like the "Transformer" proposed in the paper "Attention is all you need" by Google.

Summary

The degree of parallelization resp. the GPU acceleration of your model depends to a large extent on the architecture of the model itself.
With some architectures like RNNs parallelization is only possible to a limited degree.

Edit:

For example, if I add more layers or add more hidden nodes, should I expect GPU usage to go up?

When increasing the number of units within you should expect the GPU usage going up, matrix operations like passing an input to a hidden layer are can be well parallelized.

Adding layers is different, there you have the same problem what causes RNNs to be slow on GPU. To compute the next layer you need to have already the result of the previous layer. So you need to compute one layer after another, it's not possible to compute all at the same time.

This is the theory - In practice you might see some minor differences in GPU usage, depending on the actual implementation of the framework.
