Tensorflow 2.11 and 2.14 Memory Issue
@camda03,
Could you please provide the complete code to debug the issue in an effective way. Also   the problem might be in the format_image method. It's not really formatting the images because the transformation is assigned to a variable images and the method is returning image (the same input)
![Screenshot 2023-05-03 5 09 48 PM](https://user-images.githubusercontent.com/81610181/235905929-c4dbf2c9-0a8c-40d1-a57f-481856ad1e8b.png)

Can you try reducing batch size? This error message is generated if the memory is not sufficient to manage the batch size. Thank you!
 I'm currently uploading the code and (large) supporting data files to Dropbox.  

The normal batch size is 32.  I also tried running with 8 and 16.  Both of these failed the same way.

I'm not sure about the format image method reference.

I'll let you know when the upload is done.  It's going to take quite awhile.

What account should I open the folder to so that you can access it?

Thanks! Here are the links to the source code and supporting files.

Please let me know if you have trouble accessing, have questions, etc.

Thanks!

https://www.dropbox.com/s/ggfgh525ow3gfxd/TensorFlow%20Debug%20Files.tar.xz?dl=0
https://www.dropbox.com/s/n7zo4yrv988nc9y/TensorFlow%20Debugging%2060469.ipynb?dl=0 @camda03,
The provided links which are mentioned contain the files of 10GB and more. It would be a very difficult and tedious process for us to replicate the issue with the huge amount of data. Could you please simplify and provide the code or data or colab gist, so that we can analyse the issue in an effective way. Thank you! https://www.dropbox.com/s/3q9h77dzclg0twi/Tensorflow%2060469%20Small.tar.xz?dl=0

Here are versions of the supporting files with ten rows each in them.

Thanks! I tried running this notebook on Google Colab.  

I setup an instance with an A100 (my system has A6000s) and ran it using the ten row files.

The instance is running TensorFlow 2.12.

I got the errors listed below.

If you'd like to get access to this instance please let me know.

Hope this helps!

Thanks!




Epoch 1/100

---------------------------------------------------------------------------

ResourceExhaustedError                    Traceback (most recent call last)

[<ipython-input-43-518cfbcdf815>](https://localhost:8080/#) in <cell line: 1>()
----> 1 history = the_model.fit (training_dataset, epochs=num_epochs, validation_data=validation_dataset, callbacks=[tensorboard_cb])

1 frames

[/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/execute.py](https://localhost:8080/#) in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50   try:
     51     ctx.ensure_initialized()
---> 52     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                         inputs, attrs, num_outputs)
     54   except core._NotOkStatusException as e:

ResourceExhaustedError: Graph execution error:

Detected at node 'gradient_tape/model/layer_1/MatMul/MatMul' defined at (most recent call last):
    File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
      exec(code, run_globals)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/usr/local/lib/python3.10/dist-packages/traitlets/config/application.py", line 992, in launch_instance
      app.start()
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelapp.py", line 619, in start
      self.io_loop.start()
    File "/usr/local/lib/python3.10/dist-packages/tornado/platform/asyncio.py", line 195, in start
      self.asyncio_loop.run_forever()
    File "/usr/lib/python3.10/asyncio/base_events.py", line 603, in run_forever
      self._run_once()
    File "/usr/lib/python3.10/asyncio/base_events.py", line 1909, in _run_once
      handle._run()
    File "/usr/lib/python3.10/asyncio/events.py", line 80, in _run
      self._context.run(self._callback, *self._args)
    File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 685, in <lambda>
      lambda f: self._run_callback(functools.partial(callback, future))
    File "/usr/local/lib/python3.10/dist-packages/tornado/ioloop.py", line 738, in _run_callback
      ret = callback()
    File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 825, in inner
      self.ctx_run(self.run)
    File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 786, in run
      yielded = self.gen.send(value)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 361, in process_one
      yield gen.maybe_future(dispatch(*args))
    File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
      yielded = ctx_run(next, result)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 261, in dispatch_shell
      yield gen.maybe_future(handler(stream, idents, msg))
    File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
      yielded = ctx_run(next, result)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/kernelbase.py", line 539, in execute_request
      self.do_execute(
    File "/usr/local/lib/python3.10/dist-packages/tornado/gen.py", line 234, in wrapper
      yielded = ctx_run(next, result)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/ipkernel.py", line 302, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/usr/local/lib/python3.10/dist-packages/ipykernel/zmqshell.py", line 539, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 2975, in run_cell
      result = self._run_cell(
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3030, in _run_cell
      return runner(coro)
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/async_helpers.py", line 78, in _pseudo_sync_runner
      coro.send(None)
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3257, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3473, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "/usr/local/lib/python3.10/dist-packages/IPython/core/interactiveshell.py", line 3553, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "<ipython-input-43-518cfbcdf815>", line 1, in <cell line: 1>
      history = the_model.fit (training_dataset, epochs=num_epochs, validation_data=validation_dataset, callbacks=[tensorboard_cb])
    File "/usr/local/lib/python3.10/dist-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/local/lib/python3.10/dist-packages/keras/engine/training.py", line 1685, in fit
      tmp_logs = self.train_function(iterator)
    File "/usr/local/lib/python3.10/dist-packages/keras/engine/training.py", line 1284, in train_function
      return step_function(self, iterator)
    File "/usr/local/lib/python3.10/dist-packages/keras/engine/training.py", line 1268, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/local/lib/python3.10/dist-packages/keras/engine/training.py", line 1249, in run_step
      outputs = model.train_step(data)
    File "/usr/local/lib/python3.10/dist-packages/keras/engine/training.py", line 1054, in train_step
      self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/usr/local/lib/python3.10/dist-packages/keras/optimizers/optimizer.py", line 542, in minimize
      grads_and_vars = self.compute_gradients(loss, var_list, tape)
    File "/usr/local/lib/python3.10/dist-packages/keras/optimizers/optimizer.py", line 275, in compute_gradients
      grads = tape.gradient(loss, var_list)
Node: 'gradient_tape/model/layer_1/MatMul/MatMul'
OOM when allocating tensor with shape[443156,3700] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
	 [[{{node gradient_tape/model/layer_1/MatMul/MatMul}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_train_function_30186] @camda03,
Could you please share the colab gist for the reference where you are trying to execute the code? Thank you! https://colab.research.google.com/gist/camda03/7b93f3bcb8fc6474dc125798a2c3d034/tensorflow-debugging-60469.ipynb

Please let me know if this does or doesn't work for you or if you have questions.

Thanks! @camda03,
The reason for this is the high amount of parameters (please check your model.summary()).
You are exhausting the available memory, Could you can reduce the batch size, which would slow down the training process but let you fit the data.

And also need to serialize your data(instead of loading everything in your GPU,RAM) by converting your data to TRFRecord files:
https://www.tensorflow.org/guide/data
https://www.tensorflow.org/tutorials/load_data/tfrecord

 "The reason for this is the high amount of parameters (please check your model.summary())."
Again, this number of parameters worked just fine in TF <= 2.10.

(I have had plenty of experience figuring out just how many parameters I can get into an A6000.)

As I wrote above.
"Here is the model.  I have tried reducing the 3,700 width down to 3,400 but that doesn't help.  This exact model (3,700) works fine with <= 2.10 and has for months."

As I wrote above.
"The normal batch size is 32. I also tried running with 8 and 16. Both of these failed the same way."

In the gist I submitted, the following code appears.

with tf.device('/CPU:0'):    
    training_dataset = tf.data.Dataset.from_tensor_slices((training_df_standardized, training_labels_df)).batch(batch_size).prefetch(dataset_prefetch)
    validation_dataset = tf.data.Dataset.from_tensor_slices((validation_df_standardized, validation_labels_df)).batch(batch_size).prefetch(dataset_prefetch)
    inference_dataset = tf.data.Dataset.from_tensor_slices((inference_df_standardized)).batch(batch_size).prefetch(dataset_prefetch)

I've been putting the datasets in CPU (not GPU) ram for a long time.  This has worked well up until TF 2.11.

With 2.11 and after, this doesn't work on my A6000s or even an A100 at Google Colab.

This is all documented in the case notes to date (above).

When I look at the links you sent, I find this.

"Note: While useful, these structures are optional. There is no need to convert existing code to use TFRecords, unless you are [using tf.data](https://www.tensorflow.org/guide/data) and reading data is still the bottleneck to training. You can refer to [Better performance with the tf.data API](https://www.tensorflow.org/guide/data_performance) for dataset performance tips."

Again, the code I submitted ran just fine in TF <= 2.10.  

I would understand this better if this was some new piece of code and I was having trouble with it.  

That is not the case.

The code that ran fine in <= 2.10 on an A6000 and on a system with 1TB of CPU ram now won't run even on an A100 at Google Colab. 

What happened after TF 2.10?

Are there any plans to address this (whatever it is) in later releases of TensorFlow?

Please advise

Thanks!


 I have an update for you.

This same code runs properly using TF 2.13, and it fails using 2.14.

Here is a file showing runs with containers for 2.13 and 2.14 aka latest.

Both runs were done using the same .ipynb file, the same data, and the same GPU on the same system.

The only difference was the container and that is shown in the log.

[TensorFlow 2.13 and 2.14.txt](https://github.com/tensorflow/tensorflow/files/13126673/TensorFlow.2.13.and.2.14.txt)

The models are the same as the one I included in the gist I sent earlier.

I have a model training now using 2.13.

To summarize, a model that works on TF <= 2.10 doesn't work on 2.11, works on 2.13, and doesn't work on 2.14.
I haven't tried 2.12.

Thanks!

 The reason for this is the high amount of parameters (please check your model.summary()).
You are exhausting the available memory, Could you can reduce the batch size, which would slow down the training process but let you fit the data.

Also could you please try in the latest TensorFlow v2.16 as most of the bugs are resolved. Thank you!
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary  (Lambda Stack)

### Tensorflow Version

2.11

### Custom Code

Yes

### OS Platform and Distribution

Ubuntu 20.04

### Mobile device

_No response_

### Python version

3.8.10

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

Driver Version: 525.105.17   CUDA Version: 12.0   

### GPU model and memory

A6000    System 1TB RAM 2TB Swap

### Current Behaviour?

I have multiple models that have worked fine for months or years under version <= 2.10.

With 2.11, I get this (below).

I have (on another system) tested 2.10 vs 2.11 by switching Docker containers.  The exact same code, data, etc, works with 2.10 but fails with 2.11.


training_cycle =  0


on_train_begin lr =  1.5625000182595272e-09
Epoch 1/100

---------------------------------------------------------------------------
ResourceExhaustedError                    Traceback (most recent call last)
<ipython-input-282-89befc246e8e> in <module>
     13 
     14     if training_validation_split != None:
---> 15         history = midas_model.fit (training_dataset, epochs=num_epochs, validation_data=validation_dataset, callbacks=[tensorboard_cb, dhc_scheduler])
     16     else:
     17         history = midas_model.fit (training_dataset, epochs=num_epochs, callbacks=[tensorboard_cb, dhc_scheduler])

/usr/lib/python3/dist-packages/keras/utils/traceback_utils.py in error_handler(*args, **kwargs)
     68             # To get the full stack trace, call:
     69             # `tf.debugging.disable_traceback_filtering()`
---> 70             raise e.with_traceback(filtered_tb) from None
     71         finally:
     72             del filtered_tb

/usr/lib/python3/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     50   try:
     51     ctx.ensure_initialized()
---> 52     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
     53                                         inputs, attrs, num_outputs)
     54   except core._NotOkStatusException as e:

ResourceExhaustedError: Graph execution error:

Detected at node 'StatefulPartitionedCall' defined at (most recent call last):
    File "/usr/lib/python3.8/runpy.py", line 194, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/usr/lib/python3.8/runpy.py", line 87, in _run_code
      exec(code, run_globals)
    File "/usr/lib/python3/dist-packages/ipykernel_launcher.py", line 16, in <module>
      app.launch_new_instance()
    File "/usr/lib/python3/dist-packages/traitlets/config/application.py", line 664, in launch_instance
      app.start()
    File "/usr/lib/python3/dist-packages/ipykernel/kernelapp.py", line 583, in start
      self.io_loop.start()
    File "/usr/lib/python3/dist-packages/tornado/platform/asyncio.py", line 132, in start
      self.asyncio_loop.run_forever()
    File "/usr/lib/python3.8/asyncio/base_events.py", line 570, in run_forever
      self._run_once()
    File "/usr/lib/python3.8/asyncio/base_events.py", line 1859, in _run_once
      handle._run()
    File "/usr/lib/python3.8/asyncio/events.py", line 81, in _run
      self._context.run(self._callback, *self._args)
    File "/usr/lib/python3/dist-packages/tornado/ioloop.py", line 758, in _run_callback
      ret = callback()
    File "/usr/lib/python3/dist-packages/tornado/stack_context.py", line 300, in null_wrapper
      return fn(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1248, in inner
      self.run()
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1162, in run
      yielded = self.gen.send(value)
    File "/usr/lib/python3/dist-packages/ipykernel/kernelbase.py", line 381, in dispatch_queue
      yield self.process_one()
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 346, in wrapper
      runner = Runner(result, future, yielded)
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1095, in __init__
      self.run()
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 1162, in run
      yielded = self.gen.send(value)
    File "/usr/lib/python3/dist-packages/ipykernel/kernelbase.py", line 365, in process_one
      yield gen.maybe_future(dispatch(*args))
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 326, in wrapper
      yielded = next(result)
    File "/usr/lib/python3/dist-packages/ipykernel/kernelbase.py", line 268, in dispatch_shell
      yield gen.maybe_future(handler(stream, idents, msg))
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 326, in wrapper
      yielded = next(result)
    File "/usr/lib/python3/dist-packages/ipykernel/kernelbase.py", line 543, in execute_request
      self.do_execute(
    File "/usr/lib/python3/dist-packages/tornado/gen.py", line 326, in wrapper
      yielded = next(result)
    File "/usr/lib/python3/dist-packages/ipykernel/ipkernel.py", line 300, in do_execute
      res = shell.run_cell(code, store_history=store_history, silent=silent)
    File "/usr/lib/python3/dist-packages/ipykernel/zmqshell.py", line 536, in run_cell
      return super(ZMQInteractiveShell, self).run_cell(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 2857, in run_cell
      result = self._run_cell(
    File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 2886, in _run_cell
      return runner(coro)
    File "/usr/lib/python3/dist-packages/IPython/core/async_helpers.py", line 68, in _pseudo_sync_runner
      coro.send(None)
    File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 3062, in run_cell_async
      has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
    File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 3254, in run_ast_nodes
      if (await self.run_code(code, result,  async_=asy)):
    File "/usr/lib/python3/dist-packages/IPython/core/interactiveshell.py", line 3331, in run_code
      exec(code_obj, self.user_global_ns, self.user_ns)
    File "<ipython-input-282-89befc246e8e>", line 15, in <module>
      history = midas_model.fit (training_dataset, epochs=num_epochs, validation_data=validation_dataset, callbacks=[tensorboard_cb, dhc_scheduler])
    File "/usr/lib/python3/dist-packages/keras/utils/traceback_utils.py", line 65, in error_handler
      return fn(*args, **kwargs)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1650, in fit
      tmp_logs = self.train_function(iterator)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1249, in train_function
      return step_function(self, iterator)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1233, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1222, in run_step
      outputs = model.train_step(data)
    File "/usr/lib/python3/dist-packages/keras/engine/training.py", line 1027, in train_step
      self.optimizer.minimize(loss, self.trainable_variables, tape=tape)
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 527, in minimize
      self.apply_gradients(grads_and_vars)
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1140, in apply_gradients
      return super().apply_gradients(grads_and_vars, name=name)
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 634, in apply_gradients
      iteration = self._internal_apply_gradients(grads_and_vars)
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1166, in _internal_apply_gradients
      return tf.__internal__.distribute.interim.maybe_merge_call(
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1216, in _distributed_apply_gradients_fn
      distribution.extended.update(
    File "/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/optimizer.py", line 1211, in apply_grad_to_update_var
      return self._update_step_xla(grad, var, id(self._var_key(var)))
Node: 'StatefulPartitionedCall'
Out of memory while trying to allocate 6558708800 bytes.
BufferAssignment OOM Debugging.
BufferAssignment stats:
             parameter allocation:   18.32GiB
              constant allocation:         0B
        maybe_live_out allocation:   12.22GiB
     preallocated temp allocation:         0B
                 total allocation:   18.32GiB
Peak buffers:
	Buffer 1:
		Size: 6.11GiB
		Operator: op_name="XLA_Args"
		Entry Parameter Subshape: f32[443156,3700]
		==========================

	Buffer 2:
		Size: 6.11GiB
		Operator: op_name="XLA_Args"
		Entry Parameter Subshape: f32[443156,3700]
		==========================

	Buffer 3:
		Size: 6.11GiB
		Operator: op_name="XLA_Args"
		Entry Parameter Subshape: f32[443156,3700]
		==========================

	Buffer 4:
		Size: 16B
		Operator: op_type="AssignAddVariableOp" op_name="AssignAddVariableOp" source_file="/usr/lib/python3/dist-packages/keras/optimizers/optimizer_experimental/sgd.py" source_line=182
		XLA Label: fusion
		Shape: (f32[443156,3700], f32[443156,3700])
		==========================

	Buffer 5:
		Size: 4B
		Operator: op_name="XLA_Args"
		Entry Parameter Subshape: f32[]
		==========================


	 [[{{node StatefulPartitionedCall}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. This isn't available when running in Eager mode.
 [Op:__inference_train_function_30187]



### Standalone code to reproduce the issue

```shell
Here is the model.  I have tried reducing the 3,700 width down to 3,400 but that doesn't help.  This exact model (3,700) works fine with <= 2.10 and has for months.


Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #   
=================================================================
 layer_0 (InputLayer)        [(None, 443156)]          0         
                                                                 
 layer_1 (Dense)             (None, 3700)              1639680900
                                                                 
 layer_2 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_3 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_4 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_5 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_6 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_7 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_8 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_9 (Dense)             (None, 3700)              13693700  
                                                                 
 layer_10 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_11 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_12 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_13 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_14 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_15 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_16 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_17 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_18 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_19 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_20 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_21 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_22 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_23 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_24 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_25 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_26 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_27 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_28 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_29 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_30 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_31 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_32 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_33 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_34 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_35 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_36 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_37 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_38 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_39 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_40 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_41 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_42 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_43 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_44 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_45 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_46 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_47 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_48 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_49 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_50 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_51 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_52 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_53 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_54 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_55 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_56 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_57 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_58 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_59 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_60 (Dense)            (None, 3700)              13693700  

                                                                 
 layer_61 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_62 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_63 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_64 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_65 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_66 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_67 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_68 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_69 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_70 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_71 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_72 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_73 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_74 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_75 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_76 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_77 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_78 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_79 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_80 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_81 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_82 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_83 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_84 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_85 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_86 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_87 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_88 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_89 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_90 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_91 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_92 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_93 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_94 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_95 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_96 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_97 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_98 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_99 (Dense)            (None, 3700)              13693700  
                                                                 
 layer_100 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_101 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_102 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_103 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_104 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_105 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_106 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_107 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_108 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_109 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_110 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_111 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_112 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_113 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_114 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_115 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_116 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_117 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_118 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_119 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_120 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_121 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_122 (Dense)           (None, 3700)              13693700  
                                                                 

 layer_123 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_124 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_125 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_126 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_127 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_128 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_129 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_130 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_131 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_132 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_133 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_134 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_135 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_136 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_137 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_138 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_139 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_140 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_141 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_142 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_143 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_144 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_145 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_146 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_147 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_148 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_149 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_150 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_151 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_152 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_153 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_154 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_155 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_156 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_157 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_158 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_159 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_160 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_161 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_162 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_163 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_164 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_165 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_166 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_167 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_168 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_169 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_170 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_171 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_172 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_173 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_174 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_175 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_176 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_177 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_178 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_179 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_180 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_181 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_182 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_183 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_184 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_185 (Dense)           (None, 3700)              13693700  

                                                                 
 layer_186 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_187 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_188 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_189 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_190 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_191 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_192 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_193 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_194 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_195 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_196 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_197 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_198 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_199 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_200 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_201 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_202 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_203 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_204 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_205 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_206 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_207 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_208 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_209 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_210 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_211 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_212 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_213 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_214 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_215 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_216 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_217 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_218 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_219 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_220 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_221 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_222 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_223 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_224 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_225 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_226 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_227 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_228 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_229 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_230 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_231 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_232 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_233 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_234 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_235 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_236 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_237 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_238 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_239 (Dense)           (None, 3700)              13693700  
                                                                 
 layer_240 (Dense)           (None, 3700)              13693700  
                                                                 
 output_layer (Dense)        (None, 1)                 3701      
                                                                 
=================================================================
Total params: 4,912,478,901
Trainable params: 4,912,478,901
Non-trainable params: 0
_________________________________________________________________

​


midas_model.compile(loss=loss_list, optimizer = the_optimizer)


Here are some other potential items of interest.

the_optimizer = keras.optimizers.SGD (nesterov=True, momentum=0.9, learning_rate=initial_learning_rate)

Loss:  mae = tf.keras.losses.MeanAbsoluteError()
```


### Relevant log output

_No response_</details>