Issues in Tensorflow model training
Hi **@cupidp9** ,


When your system memory peaks, it can lead to the training process being killed to prevent the system from crashing. This is what the "^C" in your log indicates—it's the interrupt signal, typically generated by pressing Ctrl+C, but in this context, it might also indicate that the Colab environment killed your process due to excessive resource usage.

Reduce Memory Usage:

Reduce Batch Size: You have already tried reducing the batch size to 2, which is quite small, but if you are still hitting memory limits, consider if there is any other part of your pipeline that can be optimized for memory usage.
Simplify Your Model: If possible, use a simpler or smaller model architecture that requires less memory during training.

Optimize TensorFlow and GPU Usage:

Since the logs mention TensorFlow trying to find TensorRT and CUDA devices and failing, it indicates that the training is attempting to run on a GPU but cannot find one. Google Colab offers free access to GPUs and TPUs, but you need to make sure you have requested access to these resources in your notebook 
```(Runtime > Change runtime type > Hardware accelerator).```

Thank you! > Hi **@cupidp9** ,
> 
> When your system memory peaks, it can lead to the training process being killed to prevent the system from crashing. This is what the "^C" in your log indicates—it's the interrupt signal, typically generated by pressing Ctrl+C, but in this context, it might also indicate that the Colab environment killed your process due to excessive resource usage.
> 
> Reduce Memory Usage:
> 
> Reduce Batch Size: You have already tried reducing the batch size to 2, which is quite small, but if you are still hitting memory limits, consider if there is any other part of your pipeline that can be optimized for memory usage. Simplify Your Model: If possible, use a simpler or smaller model architecture that requires less memory during training.
> 
> Optimize TensorFlow and GPU Usage:
> 
> Since the logs mention TensorFlow trying to find TensorRT and CUDA devices and failing, it indicates that the training is attempting to run on a GPU but cannot find one. Google Colab offers free access to GPUs and TPUs, but you need to make sure you have requested access to these resources in your notebook `(Runtime > Change runtime type > Hardware accelerator).`
> 
> Thank you!

Thanks for your reply! Well, the size of my input images was the problem. They were too large and due to that reason the system memory was going at it's peak and stopping.  Can anyone guide me how to contribute to this open source project i am novice please provide guidance. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63103">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/63103">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.13

### Custom code

No

### OS platform and distribution

Linux Ubuntu 18.04

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

0

I am trying to run a pretrained tensorflow object detection model with my own datasets in google collab. But getting this unknown errors while running the training. I tried changing the batch size=2, and steps=20000, but still the same errors. Its just stopping the training with ^C. I noticed my system memory went its peak just before stopping. is it something due to this?



### Standalone code to reproduce the issue

```shell
!python /content/models/research/object_detection/model_main_tf2.py \
    --pipeline_config_path={pipeline_file} \
    --model_dir={model_dir} \
    --alsologtostderr \
    --num_train_steps={num_steps} \
    --sample_1_of_n_eval_examples=1
```


### Relevant log output

```shell
2024-03-01 11:15:07.851290: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-03-01 11:15:14.012057: E tensorflow/compiler/xla/stream_executor/cuda/cuda_driver.cc:268] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
INFO:tensorflow:Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
I0301 11:15:14.048205 139949003751424 mirrored_strategy.py:419] Using MirroredStrategy with devices ('/job:localhost/replica:0/task:0/device:CPU:0',)
INFO:tensorflow:Maybe overwriting train_steps: 20000
I0301 11:15:14.210126 139949003751424 config_util.py:552] Maybe overwriting train_steps: 20000
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0301 11:15:14.210465 139949003751424 config_util.py:552] Maybe overwriting use_bfloat16: False
WARNING:tensorflow:From /content/models/research/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
W0301 11:15:14.290481 139949003751424 deprecation.py:364] From /content/models/research/object_detection/model_lib_v2.py:563: StrategyBase.experimental_distribute_datasets_from_function (from tensorflow.python.distribute.distribute_lib) is deprecated and will be removed in a future version.
Instructions for updating:
rename to distribute_datasets_from_function
INFO:tensorflow:Reading unweighted datasets: ['/content/train.tfrecord']
I0301 11:15:14.314484 139949003751424 dataset_builder.py:162] Reading unweighted datasets: ['/content/train.tfrecord']
INFO:tensorflow:Reading record datasets for input file: ['/content/train.tfrecord']
I0301 11:15:14.314929 139949003751424 dataset_builder.py:79] Reading record datasets for input file: ['/content/train.tfrecord']
INFO:tensorflow:Number of filenames to read: 1
I0301 11:15:14.315054 139949003751424 dataset_builder.py:80] Number of filenames to read: 1
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0301 11:15:14.315170 139949003751424 dataset_builder.py:86] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
W0301 11:15:14.353366 139949003751424 deprecation.py:364] From /content/models/research/object_detection/builders/dataset_builder.py:100: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.AUTOTUNE)` instead. If sloppy execution is desired, use `tf.data.Options.deterministic`.
WARNING:tensorflow:From /content/models/research/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
W0301 11:15:14.419352 139949003751424 deprecation.py:364] From /content/models/research/object_detection/builders/dataset_builder.py:235: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.map()
WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
W0301 11:15:24.248110 139949003751424 deprecation.py:364] From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: sparse_to_dense (from tensorflow.python.ops.sparse_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Create a `tf.sparse.SparseTensor` and use `tf.sparse.to_dense` instead.
WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
W0301 11:15:28.897560 139949003751424 deprecation.py:364] From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: sample_distorted_bounding_box (from tensorflow.python.ops.image_ops_impl) is deprecated and will be removed in a future version.
Instructions for updating:
`seed2` arg is deprecated.Use sample_distorted_bounding_box_v2 instead.
WARNING:tensorflow:From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0301 11:15:32.042234 139949003751424 deprecation.py:364] From /usr/local/lib/python3.10/dist-packages/tensorflow/python/util/dispatch.py:1176: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
2024-03-01 11:15:35.407217: W tensorflow/core/framework/dataset.cc:956] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
^C
```
![Screenshot 2024-03-01 132219](https://github.com/tensorflow/tensorflow/assets/161838020/68815ba7-4888-4046-b0f3-5061a93ec818)

