Significant performance drop in `ModifyGraphWithDelegate() `on Android 12 Using Hexagon Delegate
Hi @Janus-Shiau 

Thanks for reporting the issue.

Could you please share your findings on latest stable version TF2.13 and nightly snapshots?

Thanks. Hi pjpratik,

We have tested both on TF2.13 and nightly snapshots. The results are as below.

tf_2.7/2.11_mobileNet_unitTest@Android10: 150 ms = 0.15 seconds
tf_2.11_mobileNet_unitTest@Android12: 4964 ms = 5 seconds
tf_2.13_mobileNet_unitTest@Android12: 3681ms = 3.7 seconds
tf_nightly_mobileNet_unitTest@Android12: 3880ms = 3.9 seconds

We also find that if we build tensorflow-lite from source with printf()-type message logs inside op_builder->PopulateSubGraph() called by the ops being built , all of the logs will be printed following the sequential order same as the code flow in the BuildGraph() function by the code running on Android 12 . However only half of the logs will be printed by the code running on Android 10, and they appear to be printed in an unordered manner. From the clues above, we suspect that the BuildGraph() function will be executed by multiple CPU threads on Android 10 but somehow only single thread on Android 12. The build id of Android 10 is “QKQ1.210528.001” and the the build id of Android 12 is “SKQ1.220804.001”

Thanks. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Hi,

Is there any update?

Thanks. Hi @sirakiin,

Can you please take a look? Attached is a project that should be close to reproducing it, but there's no emulator for Snapdragons. Thanks.

[Test61870.zip](https://github.com/tensorflow/tensorflow/files/12823349/Test61870.zip)
 Hi, is there any update here? The default number of threads has changed in different versions of TF Lite.
For reliable performance, you should explicitly set the number of threads using the `SetNumThreads` method of `InterpreterBuilder` (not to be confused with the similarly named method of `Interpreter`, which is deprecated).

Unfortunately at this point changing the default number of threads back to what it was in TensorFlow 2.7 would also cause new regressions for other apps that are relying on the new default. > The default number of threads has changed in different versions of TF Lite.

More specifically, TF Lite 2.7 used Eigen which defaulted to 4 threads, whereas later versions of TF Lite default to 1 thread.

So if your code has had a performance regression since TF Lite 2.7, and it appear to be related to threading, try calling `.SetNumThreads(4)` on your `InterpreterBuilder` instance before constructing the `Interpreter`.
### Issue type

Performance

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

Tensorflow 2.7 & 2.11

### Custom code

No

### OS platform and distribution

Linux Ubuntu 18.04

### Mobile device

Android 12

### Python version

3.9

### Bazel version

3.7.2

### GCC/compiler version

7.5.0

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

#### Significant performance drop

Using the same `libhexagon_nn_skel_v66.so`, `libhexagon_interface.so` in `v1.20.0.1` from the [official website](https://www.tensorflow.org/lite/android/delegates/hexagon).

`ModifyGraphWithDelegate()` of **hexagon delegate on Android 12 is 33x slower than the same function on Android 10**.

#### Possible bugs
We have found that the function `BuildGraph()` in the file of the path below is the source of the irregular execution time but don't know why. 

```
tensorflow_source_code_root_dir\tensorflow\lite\delegates\hexagon\hexagon_delegate_kernel.cc
```

#### The SoC

We run the code on Snapdragon XR2, but we think it's easy to reproduce it on other SoC with supported Hexagon.

### Standalone code to reproduce the issue

You can download some tflite model to reproduce this issue. for example, the [SSD-MobileNet from TFHub](https://tfhub.dev/tensorflow/lite-model/ssd_mobilenet_v1/1/default/1), can reproduce this performance drop.

```shell
std::unique_ptr<tflite::Interpreter> interpreter;
tflite::ops::builtin::BuiltinOpResolver resolver;

std::unique_ptr<tflite::FlatBufferModel> model = tflite::FlatBufferModel::BuildFromFile("/vendor/etc/ssd_mobilenet_v1_1_default_1.tflite");
InterpreterBuilder builder(*model, resolver);
builder(&interperter);

TfLiteHexagonInitWithPath("/vendor/lib/rfsa/adsp");
TfLiteHexagonDelegateOptions params = {0};
auto *delegate_ptr = TfLiteHexagonDelegateCreate(&params);
Interpreter::TfLiteDelegatePtr delegate(delegate_ptr, [](TfLiteDelegate *delegate){TfLiteHexagonDelegateDelete(delegate)});

uint64_t start_time = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();

if(interperter->ModifyGraphWithDelegate(delegate.get()) != kTfLiteOk){
        printf("Fail to delegate with Hexagon\n");
}else{
        printf("Success to delegate with Hexagon\n");
}

uint64_t end_time = std::chrono::duration_cast<std::chrono::microseconds>(std::chrono::steady_clock::now().time_since_epoch()).count();
uint64_t using_time = end_time > start_time ? end_time - start_time : 0;
printf("using_time: %u micro seconds = %f seconds\n", using_time, using_time / 1000000.f);

if(interperter->AllocateTensors() != kTfLiteOk){
        printf("Fail to allocate\n");
}
```


### Relevant log output

```shell
# On Android12->
Success to delegate with Hexagon
using_time: 4964460 micro seconds = 4.964 seconds

# On Android10->
Success to delegate with Hexagon
using_time: 150438 micro seconds = 0.15 seconds
```
