M2 GPU utilization decays from 50% to 10% in non batched inference for huggingface distilbert-base-cased
More details:

I found no evidence yet this is a heat throttling issue, 'cos after the huge drop in GPU utilization, other processes will overtake using the GPU.
I wonder what's going on? Is there any profiling tips I can do to help investigate. I am aware I can "fix" this by doing batch inferences. (or just use CPU as some may recommend for non batch inference). But seeing this GPU utilization decay is still unsettling, since this can potentially happen for a training session (which is far longer).

So I would like to know the root cause. I have experience in TF in colab (and GCP linux), and I don't quite remember seeing this sort of behavior. I understood M2 Max is new. I found out this has something to do with the variation in length of input tokens from one inference to the next. It doesn't seem to like receiving lengths that vary greatly, maybe this causes some sort of weird fragmentation in GPU memory?? Here's the code that only extract IMDB sentences that has >512 tokens. And it is able to sustain GPU utilization, with ~30it/s.

```
from transformers import AutoTokenizer, TFDistilBertForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')

from datasets import load_dataset
imdb = load_dataset('imdb')

print('starting collecting sentences with tokens >= 512')
sentences = [sentence for sentence in imdb['train']['text'] if tokenizer(sentence, truncation=True, return_tensors='tf')['input_ids'].shape[-1] >= 512]
print('finished collecting sentences with tokens >= 512')

for k, sentence in tqdm(enumerate(sentences)):
  inputs = tokenizer(sentence, truncation=True, return_tensors='tf')

  output = model(inputs).logits
  pred = np.argmax(output.numpy(), axis=1)
  
  if k % 100 == 0:
    print(f"len(input_ids): {inputs['input_ids'].shape[-1]}")

```    

print

```
7it [00:00, 31.12it/s]
len(input_ids): 512
107it [00:03, 32.38it/s]
len(input_ids): 512
...
...
3804it [02:00, 31.85it/s]
len(input_ids): 512
3904it [02:03, 32.50it/s]
len(input_ids): 512
3946it [02:04, 31.70it/s]
```

Any ideas, is there a serious bug lurking behind? If you have instructions/tips on how to debug, I would be eager to try. Hi @kechan ,

I just tried the code on Linux environment just to cross check the behaviour but got an error. Please refer to attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/0954c3cd6b8bc90db9decfd171bdfffe/60271.ipynb) and confirm whether the attached code is missing anything ? 

Since you are using tensorflow-macos with metal plugin request you to please post the issue in Apple forum also. 

Thanks! @SuryanarayanaY Thanks for looking into it. 

I went to Colab notebook. That error I believed is due to incorrect import of tqdm. It should be:

`from tqdm import tqdm`

I will modify my code snippet to include the pip and imports from your notebook

I changed it my own copy of colab notebook, and it worked. Please bear in mind this issue is not reproducible on Colab with GPU acceleration. It happens only with metal on apple silicon. 

And yes, here is the issue I logged on their forum:

https://developer.apple.com/forums/thread/727890?answerId=750313022#750313022

I am aware it may be a bit complicated to triage in lot of cases where more than one group is involved. However, if you can provide me tips, I can try debug this as far as I could, and cross-post whatever I find over with Apple. 

While I understood the problem, and know how to work around it, I felt that the Apple GPU is not fulfilling its potential, since aligning every input length to the max_len (to ensure data uniformity) may not memory-efficient. Hi @kechan,

I got a query here. I gone through the code and In the code I can't see anywhere you have used tensorflow or keras. Is that Transformers using tensorflow  at backend ? Is that the reason you posted issue here ? I am not sure whether the issue needs to be addressed here ? Please confirm the dependency with Tensorflow/keras. Thanks!
 @SuryanarayanaY This is the transformers module from HuggingFace. It sort of a higher level framework that sits atop both Tensorflow and Pytorch. The code snippet given is backed by TF, as indicated by the model name “ TFDistilBertForSequenceClassification”. While this code is only for inference, I can also reproduce this issue during training, (i.e. calling model.fit) @kechan ,

Thanks for the confirmation.You mean that `TFDistilBertForSequenceClassification` is completely built upon Tensorflow as backend right? 

AFAIK, This is the problem with only Mac M2 as you confirmed its fine on Colab/Linux, this needs to be addressed by Apple and I see you have already logged a ticket there. I can see some response there and hope it will be addressed

If the problem also observed with Linux/colab then we might also have a look into it.I see this is not the case here.
 @SuryanarayanaY This issue is specific to Apple M2 Max with Metal (GPU). And yes, TFDistil**** means it is backed by tensorflow in the backend. When I get the chance, I will attempt to build a case where huggingface is not involved, and see if it can reproduce.  @kechan ,
Thanks for confirmation. But we are not supporting `tensorflow-macos` and it is not same as `tensorflow`. If you can reproduce the behaviour with only `tensorflow` involved then we will definitely have a look into the issue. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60271">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60271">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

tensorflow-macos 2.9, tensorflow-metal 0.5.0

### Custom Code

Yes

### OS Platform and Distribution

MacOS 13.3

### Mobile device

_No response_

### Python version

Python 3.10.9

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

N/A

### GPU model and memory

Apple M2 Max (unified memory)

### Current Behaviour?

```shell
MacBook Pro M2 Max 96gb 
macOS 13.3 
tensorflow-macos 2.9.0 
tensorflow-metal 0.5.0

GPU utilization should hold steady when running inference for HuggingFace TFDistilBertForSequenceClassification from pretrained 'distilbert-base-cased'.
Instead utilization dropped steadily from 50% to 10% (and sometimes, below 2%). 
It becomes excruciatingly slow.
```


### Standalone code to reproduce the issue

```shell
# if needed, from HuggingFace
!pip install transformers
!pip install datasets

from transformers import AutoTokenizer, TFDistilBertForSequenceClassification
from datasets import load_dataset
from tqdm import tqdm

imdb = load_dataset('imdb')
sentences = imdb['train']['text'][:500]

tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")
model = TFDistilBertForSequenceClassification.from_pretrained('distilbert-base-cased')

for i, sentence in tqdm(enumerate(sentences)):
  inputs = tokenizer(sentence, truncation=True, return_tensors='tf')
  output = model(inputs).logits
  pred = np.argmax(output.numpy(), axis=1)

  if i % 100 == 0:
    print(f"len(input_ids): {inputs['input_ids'].shape[-1]}")
```


### Relevant log output

```shell
Output from print:

Metal device set to: Apple M2 Max

systemMemory: 96.00 GB
maxCacheSize: 36.00 GB

3it [00:00, 10.87it/s]
len(input_ids): 391
101it [00:13,  6.38it/s]
len(input_ids): 215
201it [00:34,  4.78it/s]
len(input_ids): 237
301it [00:55,  4.26it/s]
len(input_ids): 256
401it [01:54,  1.12it/s]
len(input_ids): 55
500it [03:40,  2.27it/s]
```
</details>