I'm trying to calculate the amount of memory needed by a GPU to train my model based on this notes from Andrej Karphaty:http://cs231n.github.io/convolutional-networks/#computational-considerationsMy network has 532,752 activations and 19,072,984 parameters (weights and biases). These are all 32 bit floats values, so each takes 4 bytes in memory.My input image is 180x50x1 (width x height x depth) = 9,000 float 32 values. I don't use image augmentation, so I think the miscellaneous memory would be only related to the mini-batch size. I'm using a mini-batch size of 128 images.Based on Andrej's recommendation, I get the following memory sizes:Activations: 532,752*4/(1024^2) = 2.03 MBParameters: 19,072,984*4/(1024^2) * 3 = 218.27 MBMiscellaneous: 128*9,000*4/(1024^2) = 4.39 MBSo the total memory to train this network would be 224,69 MB.I'm using TensorFlow and I think I'm missing something. I haven't run the training yet, but I'm pretty sure (based on past experiences) that the memory in use will be much higher than what I've calculated.If for each image in the mini-batch, TensorFlow keeps their gradients so it can normalize them later for a single weights/biases updates step, then I think the memory should take into account another 532,752 * 128 values (gradients for each image in the mini-batch). If that is the case, then I'd need more 260.13 MB to train this model with 128 images/mini-batch.Can you help me understand the memory considerations for training my deep learning model? Are the above considerations right?
How to calculate the mini-batch memory impact when training deep learning models?
deep learning;tensorflow