title: tensorflow gpu Resource Exhausted during model definition
body: <p>I get <strong>ResourceExhaustedError</strong> from tensorflow not during training but during my model definition, so che classic suggestion &quot;decrease batch size&quot; do not makes sense in this case.</p>
  <p>Here is my model definition:</p>
  <pre><code>def build_net(input_shape=(400, 300, 3),output_class_count=6):
      model=keras.Sequential([
          layers.Input(shape=input_shape,name='Input'),
          layers.Conv2D(filters=128, kernel_size=3, strides=1,activation='relu',padding='same',name='C5'),
          layers.Conv2D(filters=64, kernel_size=3, strides=1,activation='relu',padding='same',name='C7'),
          layers.MaxPooling2D(pool_size=3, strides=2,name='MP8'),
          layers.Flatten(),
          layers.Dense(512, activation='relu',name='F10'),
          layers.Dropout(rate=0.5),
          layers.Dense(units=output_class_count,activation='softmax',name='Output')])
      return model
  
  model=build_net()
  </code></pre>
  <p>I have already used this code:</p>
  <pre><code>gpu_devices = tf.config.experimental.list_physical_devices('GPU')
  for device in gpu_devices:
      tf.config.experimental.set_memory_growth(device, True)
  </code></pre>
  <p>My <code>nvidia-smi</code> output is:</p>
  <p><a href="https://i.stack.imgur.com/OOnF3.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/OOnF3.png" alt="enter image description here" /></a></p>
  <p>So the memory is not even completely full.</p>
  <p>Here are the stats of the error.</p>
  <pre><code>Limit:                     10117251072
  InUse:                      7773144576
  MaxInUse:                   7773144832
  NumAllocs:                          22
  MaxAllocSize:               3886415872
  Reserved:                            0
  PeakReserved:                        0
  LargestFreeBlock:                    0
  </code></pre>
  <p>Here is the Error:</p>
  <pre><code>/tmp/ipykernel_19436/1499846676.py in build_mini_alexnet(input_shape, output_class_count)
       14         layers.Dense(512, activation='relu',name='F10'),
       15         layers.Dropout(rate=0.5),
  ---&gt; 16         layers.Dense(units=output_class_count,activation='softmax',name='Output')])
       17     return model
  
  /python3.7/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
      585     self._self_setattr_tracking = False  # pylint: disable=protected-access
      586     try:
  --&gt; 587       result = method(self, *args, **kwargs)
      588     finally:
      589       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access
  
  /python3.7/site-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
       65     except Exception as e:  # pylint: disable=broad-except
       66       filtered_tb = _process_traceback_frames(e.__traceback__)
  ---&gt; 67       raise e.with_traceback(filtered_tb) from None
       68     finally:
       69       del filtered_tb
  
  /python3.7/site-packages/keras/backend.py in random_uniform(self, shape, minval, maxval, dtype)
     1920     return tf.random.uniform(
     1921         shape=shape, minval=minval, maxval=maxval, dtype=dtype,
  -&gt; 1922         seed=self.make_legacy_seed())
     1923 
     1924   def truncated_normal(self, shape, mean=0., stddev=1., dtype=None):
  
  ResourceExhaustedError: failed to allocate memory [Op:AddV2]
  </code></pre>
  <p>Thank you for your help.</p>