TensorFlow profiler running into OOM issue on GPU
@ndeepesh,
Could you please provide the complete standalone code to reproduce the issue and it helps us to analyse the issue in an effective way. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Hi @tilakrayal 
Any model on GPU with above code should be good for reproducing the issue. Is there a sampling parameter that can be introduced so that not all events are captured? @ndeepesh,
Without the reproducible code, it would be difficult for us to debug the issue. In order to expedite the trouble-shooting process, could you please provide a minimal code snippet you are using. Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61526">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61526">No</a>

### Issue type

Support

### Have you reproduced the bug with TensorFlow Nightly?

Yes

### Source

source

### TensorFlow version

TF 2.11, TF 2.4

### Custom code

No

### OS platform and distribution

Red Had

### Mobile device

_No response_

### Python version

3.10

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.2

### GPU model and memory

_No response_

### Current behavior?

Running TensorFlow profiler for longer than 10 second period results into OOM error, crashes the tf inference process and the profiler returns DEADLINE_EXCEEDED. Is there anyway to limit the sampling rate or way to reduce the amount of information being collected to avoid crashing the process?

Here is the code that I run:
tensorflow_profiler.experimental.client("grpc://localhost:3222", "profiles", 30000)

### Standalone code to reproduce the issue

```shell
tensorflow_profiler.experimental.client("grpc://localhost:3222", "profiles", 30000)
```


### Relevant log output

```shell
DEADLINE_EXCEEDED
```
