title: Memory usage of neural network, Keras
body: <p>I am trying to develop a model for denoising images. I've been reading up on how to calculate memory usage of a neural network and the standard approach seems to be:</p>
  
  <pre><code>    params = depth_n x (kernel_width x kernel_height) x depth_n-1 + depth
  </code></pre>
  
  <p>By summing all parameters together in my network, I end up getting <strong>1,038,097</strong> which approximates to <strong>4.2MB</strong>. It seems I have done a slight miscalculation in the last layer since Keras ends up getting <strong>1,038,497</strong> params. Nevertheless, this is a small difference. 4.2MB is just the parameters, and I've seen somewhere that one should <strong>multiply by 3</strong> to include backprop and other needed calculations. This would then approximate to 13MB.</p>
  
  <p>I have approximately 11 GB of GPU memory to work with, yet this model gets exhausted. Where does all the extra needed memory come from? What am I missing? I know this post might be labeled as duplicate, but none of the others seems to catch the topic which I am asking about.</p>
  
  <p>My model:</p>
  
  <pre><code>    def network(self):
          weights = RandomUniform(minval=-0.05, maxval=0.05, seed=None)
          input_img = Input(shape=(self.img_rows, self.img_cols, self.channels))
          conv1 = Conv2D(1024, (3,3), activation='tanh', kernel_initializer=weights,
                  padding='same', use_bias=True)(input_img)
          conv2 = Conv2D(64, (3,3), activation='tanh', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv1)
          conv3 = Conv2D(64, (3,3), activation='tanh', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv2)
          conv4 = Conv2D(64, (3,3), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv3)
          conv5 = Conv2D(64, (7,7), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv4)
          conv6 = Conv2D(64, (5,5), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv5)
          conv7 = Conv2D(32, (5,5), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv6)
          conv8 = Conv2D(32, (3,3), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv7)
          conv9 = Conv2D(16, (3,3), activation='relu', kernel_initializer=weights,
                  padding='same', use_bias=True)(conv8)
          decoded = Conv2D(1, (5,5), kernel_initializer=weights,
                  padding='same', activation='sigmoid', use_bias=True)(conv8)
          return input_img, decoded
  
  
      def compiler(self):
          self.model.compile(optimizer='RMSprop', loss='mse')
          self.model.summary()
  </code></pre>
  
  <p>I assume my model is silly in a lot of ways and that there are multiple things to improve (dropout, other filter sizes and numbers, optimizers etc.) and all suggestions are received gladly, but the actual question still remain. Why does this model consume so much memory? Is it due to the extremely high depth of <code>conv1</code>?</p>
  
  <p>Model summary:</p>
  
  <pre><code>    Using TensorFlow backend.
      _________________________________________________________________
      Layer (type)                 Output Shape              Param #   
      =================================================================
      input_1 (InputLayer)         (None, 1751, 480, 1)      0         
      _________________________________________________________________
      conv2d_1 (Conv2D)            (None, 1751, 480, 1024)   10240     
      _________________________________________________________________
      conv2d_2 (Conv2D)            (None, 1751, 480, 64)     589888    
      _________________________________________________________________
      conv2d_3 (Conv2D)            (None, 1751, 480, 64)     36928     
      _________________________________________________________________
      conv2d_4 (Conv2D)            (None, 1751, 480, 64)     36928     
      _________________________________________________________________
      conv2d_5 (Conv2D)            (None, 1751, 480, 64)     200768    
      _________________________________________________________________
      conv2d_6 (Conv2D)            (None, 1751, 480, 64)     102464    
      _________________________________________________________________
      conv2d_7 (Conv2D)            (None, 1751, 480, 32)     51232     
      _________________________________________________________________
      conv2d_8 (Conv2D)            (None, 1751, 480, 32)     9248      
      _________________________________________________________________
      conv2d_10 (Conv2D)           (None, 1751, 480, 1)      801       
      =================================================================
      Total params: 1,038,497
      Trainable params: 1,038,497
      Non-trainable params: 0
      _________________________________________________________________
  </code></pre>