Where is the majority of Tensorflow memory usage?
Assuming your samples have a fixed size and you use constant batch size, you should never run out of memory - that is: as soon as you have managed to train a full batch (forward and backward propagation complete) you should be fine to assume it will train until completion!



Extra info / tips

The total memory requirement will be a product of the size of the model (number of parameters) along with the batch size. Each single sample that gets sent through the network will effectively influence each of the $N$ weights. A batch of size $B$ will mean that there needs to be roughly $N \times B$ memory available.

If you model is already fixed, you can start with a large batch size, decreasing it slowly until you no longer receive an outOfMemory error. You could also try computing exactly what your memory requirement are, but there are some other factors that might make that a little difficult, so it is faster to try the first approach.

If you are using a GPU, you ca look at the output of the terminal command nvidia-smi, you can see the available memory of the GPUs. You will notice it essentially becomes all used as soon as training begins. This is because Tensorflow, by default, will occupy all available memory. There are ways around that, so search for allow_growth in the Tensorflow docs for relevant information and configuration tactics.