title: TensorFlow: How to measure how much GPU memory each tensor takes?
body: <p>I'm currently implementing <a href="http://pjreddie.com/media/files/papers/yolo.pdf" rel="noreferrer">YOLO</a> in TensorFlow and I'm a little surprised on how much memory that is taking. On my GPU I can train YOLO using their <a href="http://pjreddie.com/darknet/" rel="noreferrer">Darknet</a> framework with batch size 64. On TensorFlow I can only do it with batch size 6, with 8 I already run out of memory. For the test phase I can run with batch size 64 without running out of memory.</p>
  
  <ol>
  <li><p>I am wondering how I can calculate how much memory is being consumed by each tensor? Are all tensors by default saved in the GPU? Can I simply calculate the total memory consumption as the shape * 32 bits?</p></li>
  <li><p>I noticed that since I'm using momentum, all my tensors also have a <code>/Momentum</code> tensor. Could that also be using a lot of memory?</p></li>
  <li><p>I am augmenting my dataset with a method <code>distorted_inputs</code>, very similar to the one defined in the <a href="https://tensorflow.googlesource.com/tensorflow/+/master/tensorflow/models/image/cifar10/cifar10.py" rel="noreferrer">CIFAR-10 tutorial</a>. Could it be that this part is occupying a huge chunk of memory? I believe Darknet does the modifications in the CPU.</p></li>
  </ol>