How to calculate the mini-batch memory impact when training deep learning models?
@StatsSorceress TL;DR:

I'm going through this activity to see if I can calculate the memory required myself:

Activations: 532,752 * 2 * 4 / (1024^2) = 4.06 MB

Parameters: 19,072,984 * 4 / (1024^2) * 3 = 218.27 MB

Miscellaneous: 128 * 9,000 * 4 / (1024^2) = 4.39 MB

Total Memory: (4.06 * 128) + 218.27 + 4.39 = 742.34 MB

(Someone please correct me on this if I'm wrong. FYI, you already multiplied miscellaneous by 128, so that's why I didn't multiply it by 128 above)



I would point you to this article and the corresponding video. They helped me to understand what is going on a lot better.

NOTE: The memory required to use a network for predictions is far less than that required for training for two reasons:


When predicting, we only send an image forward through the network and not backward (so we don't multiply memory X 3; see below)
There's one prediction per image (so we don't need to multiply the memory required for one image by a batch size because we don't use batches in prediction).


Process (Memory to Train)


Calculate the memory required to train on one image
Multiply this number by the number of images in your batch 


(REMEMBER: Mini-batching says we take a subset of our data, compute the gradients and errors for each image in the subset, then average these and step forward in the direction of the average. For convnets, weights and biases are shared, but the number of activations is mutliplied by the number of images in the batch.).

STEP 1: Memory for 1 Image

To train one image, you must reserve memory for:


Model Parameters:

The weights and biases at each layer, their gradients, and their momentum variables (if Adam, Adagrad, RMSProp, etc., optimizers are used)

To approximate the memory for this, calculate the memory required to store the weights and biases and multiply that by 3 (i.e. "by 3" because we're saying the amount of memory needed to store the weights and biases is (roughly) equal to that needed for the gradients and for the momentum variables)

EQUATIONS:

Convolutions:

weights(n) = depth(n) * (kernel_width * kernel_height) * depth(n-1)

biases(n) = depth(n)

Fully Connected (Dense) Layers:

weights(n) = outputs(n) * inputs(n)

biases(n) = outputs(n)


where n is the current layer and n-1 is the previous layer, and outputs are the number of outputs from the FC layer and inputs are the number of inputs to the FC layer (if the previous layer is not a fully-connected layer, the number of inputs is equal to the size of that layer flattened).

NOTE: The memory for the weights and biases alone, plus the memory for the activations for one image (see below), is the total amount of memory you need for predictions (excluding some overhead for memory for convolutions and some other things).


Activations (these are "Blobs" in Caffe):


(I'm using terms loosely here, bear with me)

Each convolution in a convolution layer produces "number of pixels in image" activations (i.e. you pass an image through a single convolution, you get a single feature map consisting of "m" activations, where "m" is the number of pixels from your image/input).

For fully-connected layers, the number of activations you produce is equal to the size of your output.

Convolutions:

activations(n) = image_width * image_height * image_num_channels

Fully Connected (Dense) Layers:

activations(n) = outputs(n)

Note that your input is really only an image at the beginning of the network. After convolutions, it turns into something else (feature maps). So really replace "image_width", "image_height", and "image_num_channels" with "input_width", "input_height", and "layer_depth" to be more precise. (It's just easier for me to think of this concept in terms of images.)

Since we also need to store the error for the activations at each layer (used in the backward pass), we multiply the number of activations by 2 to get the total number of entities we need to make room for in our storage space. The number of activations increases with the number of images in the batch, so you multiply this number by the batch size.

STEP 2: Memory to Train Batch

Sum the number of weights and biases (times 3) and the number of activations (times 2 times the batch size). Multiply this by 4, and you get the number of bytes required to train the batch. You can divide by 1024^2 to get the answer in GB.