TensorFlow profiler running into OOM issue on GPU
@rahul-fnu 
In order to expedite the trouble-shooting process, please provide a code snippet to reproduce the issue reported here. Thank you! @sushreebarsa This is exactly what I run after importing tensorflow: `tensorflow_profiler.experimental.client("grpc://localhost:3222", "profiles", 30000)` Reduce Input Size Is there a way to configure the profiler? Either by reducing the amount of information being collected or by reducing the sampling rate? @rahul-fnu Please refer to [this](https://www.tensorflow.org/tensorboard/tensorboard_profiling_keras) guide to know the way to configure the TF profiler ? Please make sure to use the latest TF version and TF profiler. 
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. @sushreebarsa @Divin-18 
This profiler is used for debugging production issues. We cant reduce the input size. Is there a sampling parameter which we can use so that not all events are captured? because of tracing and profiling, the amount of data accumulated is proportionate to the time of profile.
I would suggest that you profile a small period of time, and check if kernel or host side info is more verbose, this is really more work load specific properties.
if host side info is verbose and not very useful, you can specify host_trace_level = 0 (I am vague on if this is available on 2.11) This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61512">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61512">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

binary

### TensorFlow version

2.11.0.5

### Custom code

No

### OS platform and distribution

Linux CentOS 7.9.2009

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

Running TensorFlow profiler for longer than 10 second period crashes the inference process because of OOM error and the profiler returns DEADLINE_EXCEEDED. Is there anyway to limit the sampling rate or way to reduce the amount of information being collected to avoid crashing the process?

### Standalone code to reproduce the issue

```shell
`tensorflow_profiler.experimental.client("grpc://localhost:3222", "profiles", 30000)`
```


### Relevant log output

_No response_