TFLite - Cannot execute with NNAPI on Samsung Galaxy Tab S9 (Snapdragon 8 Gen 2). On delegate CPU available
Hi @FrancoisDn 

Sorry for the delayed response. 

Can you please try in latest nightly snapshot and see if you are facing the issue?

Also, could you please provide the Android SDK and NDK versions being used to better understand the issue?

As per the [documentation](https://www.tensorflow.org/lite/android/delegates/nnapi#enable_nnapi_cpu_implementation),
>A graph that can't be processed completely by an accelerator can fall back to the NNAPI CPU implementation. However, since this is typically less performant than the TensorFlow interpreter, this option is disabled by default in the NNAPI delegate for Android 10 (API Level 29) or above. To override this behavior, set setUseNnapiCpu to true in the NnApiDelegate.Options object.

Thanks.

    Hi @pjpratik,
thank you for you reply.
I use Android NDK 25.1.8937393 and Android SDK 33.

Regarding your remark, it's interesting, but strange for my case. In C++, I think that the equivalent to `setUseNnapiCpu()` is attribute `disallow_nnapi_cpu` of struct `tflite::StatefulNnApiDelegate::Options`. Indeed, this attribute is set `true` by default, and I do not change in my code, so the fallback to NNAPI CPU should not be used.
But according to the TFLite log, it's the case! I have no explanation for that yet.

About testing a newest version, indeed it's a very good idea. I thought about that, because I used version 2.12.0. I'm using Conan package system to easily get TFLite, but unfortunately the last stable version 2.13.0 is not yet available on Conan Center.
So I will try to get the version by hand to test if the problem is still present with 2.13.0 or the nightly build, but it will require more time than using conan.    Hi @pjpratik,
I just made a test with Tensorflow Lite last stable version 2.14.0 just released, and I have the exact same issue.
The NNAPI CPU delegate is used instead of the dedicated GPU/TPU on the Samsung S9.

_Note that I used a Int8 model (because if float32, for the test I made, they will be executed on CPU, because seem not supported by GPU/TPU with NAPI)._ Hi @FrancoisDn 

Thanks for the information.

@pkgoogle Could you please look at this issue?

Thanks. Hi @FrancoisDn, is it possible for you to share your model file to help us reproduce the issue? (If you can't share a full model, can you reduce/use a smaller/toy model to reproduce the issue?)    Hi @pkgoogle,
thank you for proposition.

I just succeed to reproduce the problem with a model not confidential.

I used a public yolov8n model : [yolov8n-I8-256x320.zip](https://github.com/tensorflow/tensorflow/files/12784276/yolov8n-I8-256x320.zip). This model was export from https://hub.ultralytics.com/models/vY89GpzxNSze7BxrYy72?tab=deploy in TFLite format with option `INT8 Quantization` and image Size `256x320`.

With that model here the behavior I have: 
 - (OK) On Samsung Galaxy S8, the inference is done on GPU/TPU. I have this log:
```Initialized TensorFlow Lite runtime.
Created TensorFlow Lite delegate for NNAPI.
NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
Replacing 253 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 1 partitions.
```

- (NOK) On Samsung S9 and also new, on Google Pixel Tab with that yolo model, the inference is done on the CPU. I have this log on S9:
```
Initialized TensorFlow Lite runtime.
Created TensorFlow Lite delegate for NNAPI.
Created TensorFlow Lite XNNPACK delegate for CPU.
Replacing 169 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 103 partitions.
```
And this log on Google Pixel Tab:
```
Initialized TensorFlow Lite runtime.
Created TensorFlow Lite delegate for NNAPI.
NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
Replacing 30 node(s) with delegate (TfLiteNnapiDelegate) node, yielding 6 partitions.
NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
Created TensorFlow Lite XNNPACK delegate for CPU.
Replacing 145 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 98 partitions.
```

As we can see, on S9 and Google Pixel Tab, even If I ask to use a NNAPI delegate to run on GPU/TPU, in both case it's run on CPU (log `Created TensorFlow Lite XNNPACK delegate for CPU`).
I can also confirmed that the inference time on Samsung S9 is slower (20 ms) than the time on S8 (14ms). So it confirmed the TFlite log which tell that NNAPI CPU is used. @Wawha, Thanks for the additional information, @miaout17, can you please take a look? Thanks. I saw similar things galaxy s23 phone.
It also has snapdragon 8 gen 2 SoC.
Any progress on this issue? Hi @miaout17 and @pkgoogle,
is there some news/progress about problem?
Can you confirmed that it's a bug with TensorFlow Lite and Galaxy Tab S9/Snapdragon 8 Gen 2 ? I saw similar issue by use public model [mobilenet_v2_1.0_224.tflite](https://commondatastorage.googleapis.com/chromeos-localmirror/distfiles/ml-test-assets-0.0.6.tar.gz) on Pixel 7 pro.
I use [TFLite Model Benchmark Tool](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/lite/tools/benchmark) to inference.

command:
adb shell /data/local/tmp/benchmark_model --verbose=true --graph=/data/local/tmp/mobilenet_v2_1.0_224.tflite --use_nnapi=true

message:
INFO: Loaded model /data/local/tmp/mobilenet_v2_1.0_224.tflite
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for NNAPI.
INFO: NNAPI delegate created.
WARNING: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
INFO: **_Though NNAPI delegate is explicitly applied, the model graph will not be executed by the delegate._**
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
VERBOSE: Replacing 66 out of 66 node(s) with delegate (TfLiteXNNPackDelegate) node, yielding 1 partitions for the whole graph. not sure whether it helps to solve the problem, but there are some discussions about Qualcomm disabling NNAPI support for Snapdragon 8 Gen 2:
https://ai-benchmark.net/index.php?threads/what-are-qualcomm-qnn-htp-dsp-delegates.44/ Hi there! Any updates on this issue?
I've been trying to run the sample model from https://www.tensorflow.org/lite/android/delegates/nnapi#use_supported_models_and_ops on the Samsung Galaxy S23 Ultra.
However, I'm encountering a similar result. I've tried benchmark tools and the sample app with the NnApiDelegate, but in all cases, I see a full fallback on CPU. There are no signs of using NNAPI.
 I'm also getting a similar issue on a Pixel 7. Unfortunately I can't share the model but the error when running the benchmarking tool is as follows:
```
INFO: Initialized TensorFlow Lite runtime.
INFO: Created TensorFlow Lite delegate for NNAPI.
WARNING: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
WARNING: NNAPI SL driver did not implement SL_ANeuralNetworksDiagnostic_registerCallbacks!
ERROR: Failed to add NN API tensor: type INT64 is not supported.
ERROR: Restored original execution plan after delegate application failure.
ERROR: Failed to apply NNAPI delegate.
ERROR: Benchmarking failed.
```
Any help much appreciated.
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.12.0

### Custom code

No

### OS platform and distribution

Android 13

### Mobile device

Samsung Galaxy Tab S9

### Python version

_No response_

### Bazel version

_No response_

### GCC/compiler version

clang 14

### CUDA/cuDNN version

_No response_

### GPU model and memory

Snapdragon 8 Gen 2

### Current behavior?

I have a C++/Kotlin Android application which make inference using Tensorflow Lite (TFLite library 2.12.0 get using conan center).
I use NNAPI delegate and it work great on Google Pixel tab and Samsung Galaxy Tab S8.

I try my application on the new tablet, Samsung Galaxy Tab S8 which have a processor 'snapdragon 8 gen 2', and this time, the inference work, BUT it use delegate for CPU and its much slower than S8...

Looking at the problem, when I call the function to build the interperter, I have this message: `INFO: Created TensorFlow Lite delegate for NNAPI.`, so everything since ok until that.

But, when I call `AllocateTensors()` on the interpreter, I have this messages:
```
Access denied finding property "ro.mediatek.platform"
Created TensorFlow Lite XNNPACK delegate for CPU.
INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
```

Is there something special to do to used NNAPI on GPU/TPU in that case?

(You can see the full output of the build of interpreter + call to AllocateTensors() below)

### Standalone code to reproduce the issue

```shell
// Load the model
std::unique_ptr<tflite::FlatBufferModel> model = tflite::FlatBufferModel::BuildFromFile(filename);

// Build the interpreter
tflite::ops::builtin::BuiltinOpResolver resolver;
std::unique_ptr<tflite::Interpreter> interpreter;
tflite::InterpreterBuilder(*model, resolver)(&interpreter);

tflite::StatefulNnApiDelegate::Options options;
tflite::StatefulNnApiDelegate delegate(options);
interpreter->ModifyGraphWithDelegate(&delegate);

// Resize input tensors -> Here the delegate CPU will be used
interpreter->AllocateTensors();
```
```


### Relevant log output

```shell
12329-12329 Manager         I  findAvailableDevices
10477-10477 tflite          I  Created TensorFlow Lite delegate for NNAPI.
10477-10584 ....C++-Error   I  INFO: Created TensorFlow Lite delegate for NNAPI.
10477-10477 libc            W  Access denied finding property "ro.mediatek.platform"
10477-10477 tflite          I  Created TensorFlow Lite XNNPACK delegate for CPU.
10477-10584 ....C++-Error   I  INFO: Created TensorFlow Lite XNNPACK delegate for CPU.
```
```
