Internal error: Error applying delegate when trying to use TensorFlow Lite NNAPI delegate on Google Pixel 7
Hi @soupslurpr 

Could you please try using the nightly snapshot and let us know if you are still facing the issue?

Thanks. Hello, thanks for responding. I just tried version 0.0.0-nightly-SNAPSHOT and the error still occurs. Hi @soupslurpr, There is a similar issue here: https://github.com/tensorflow/tensorflow/issues/61095, can you try 2.11 or 2.10 for now? Hello @pkgoogle, I just tried with both 2.11.0 and 2.10.0, and it didn't work, same error. Also tried with 2.0.0. Hi @soupslurpr, can you please let us know the minimum SDK (API Level) you are targeting? Also if you are comfortable exporting your Android project or a toy version which reproduces this issue that'll be extremely helpful! Including the .tflite model would be easiest, otherwise letting us know how you got it could also work. @pkgoogle The minimum SDK is 28 and target SDK is 33. Here is the project that reproduces this issue: https://github.com/soupslurpr/NNAPI-whisper-test Also, I got the .tflite from running this google colab: https://colab.research.google.com/github/usefulsensors/openai-whisper/blob/main/notebooks/generate_tflite_from_whisper.ipynb Hi @soupslurpr, I'm having issues opening the Android Studio project (hangs on loading), what platform are you using to open/edit your project? (MacOS, Windows, or *nix?), are there any dependencies you think I may need to install (Such as Java SDK version?) > Hi @soupslurpr, I'm having issues opening the Android Studio project (hangs on loading), what platform are you using to open/edit your project? (MacOS, Windows, or *nix?), are there any dependencies you think I may need to install (Such as Java SDK version?)

I'm running Fedora, its using Java 17, that is probably why. I'm also using the latest alpha of Android Studio which also seems to be why it isn't working for you. You can try using the latest alpha of it, or use the below stable version. 

I made a version that is compatible with the latest stable version of Android Studio here: https://github.com/soupslurpr/nnapi-whisper-test-android-studio-stable Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61126">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61126">No</a>
 Misclick, oops It seems this project requires a real device or a machine which supports VT-x or SVM.

Hi @sirakiin, can you please take a look? Thanks! @soupslurpr if you haven't figured it out yet, the tflite model file you use is the culprit and It's a [weights-only quantized one](https://www.tensorflow.org/lite/performance/post_training_quant), which doesn't work with NNAPI delegate.  > @soupslurpr if you haven't figured it out yet, the tflite model file you use is the culprit and It's a [weights-only quantized one](https://www.tensorflow.org/lite/performance/post_training_quant), which doesn't work with NNAPI delegate.

Ah, I see. Thanks for informing me of this. Do you know of a way to make it compatible? > @soupslurpr if you haven't figured it out yet, the tflite model file you use is the culprit and It's a [weights-only quantized one](https://www.tensorflow.org/lite/performance/post_training_quant), which doesn't work with NNAPI delegate.

i have this problem with whisper-tiny.tflite model but i use many other models like this one:
https://tfhub.dev/tulasiram58827/lite-model/hifi-gan/float16/1
and still get this error without any success :     
Caused by: java.lang.IllegalArgumentException: Internal error: Error applying delegate: 
        at org.tensorflow.lite.NativeInterpreterWrapper.createInterpreter(Native Method)
        
and my code simply is:
NnApiDelegate nnApiDelegate = new NnApiDelegate();
Interpreter.Options options = (new Interpreter.Options()).addDelegate(nnApiDelegate);
Interpreter  mInterpreter = new Interpreter(tfliteModel,options); <-- where Error shows up    I meant to comment on this previously, but forgot. I've also been working with a TFLite model of Whisper and there are problems beyond quantization. If using the one from https://github.com/nyadla-sys/whisper.tflite, many of the ops simply aren't supported for the GPU delegate.

Some of these are the control flow ops in the decoding process (such as `If` and `While`). None of these types of control flow ops are supported by the GPU delegate (or the NNAPI version either probably). Additionally, there are computation ops that aren't supported either by the delegate (I forget it at the moment, but I believe it is something like `NDScatterAndUpdate` that causes problems).

Regardless, without modifying the graph definition to only supported ops, how it was quantized won't matter. @mitchelldehaven how would you go about creating a tiny model that supports GPU or NNAPI? > @mitchelldehaven how would you go about creating a tiny model that supports GPU or NNAPI?

You would need to do a couple things:

- First, you would need to separate the encoder and decoder and re-implement the control flow in whatever language you are using (presumably Java if you are using NNAPI). 
- Second, the particular op `NDScatterAndUpdate` is not supported. You would need to re-implement this functionality with supported ops (presumably possible, I'm not sure exactly how).
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

tensorflow-lite 2.12.0

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

Google Pixel 7

### Python version

_No response_

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

When I try to use the NNAPI delegate to run the OpenAI whisper model on a Pixel 7, it crashes when trying to initialize the TFLite interpreter.

### Standalone code to reproduce the issue

```shell
MainActivity.kt
class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        setContent {
            NNAPIWhisperTestTheme {
                // A surface container using the 'background' color from the theme
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = MaterialTheme.colorScheme.background
                ) {
                    Test(applicationContext)
                }
            }
        }
    }
}

@Composable
fun Test(applicationContext: Context, modifier: Modifier = Modifier, viewModel: NnApiViewModel = viewModel()) {
    Scaffold { innerPadding ->
        Column {
            Button(modifier = Modifier.padding(innerPadding), onClick = { viewModel.initialize(applicationContext) }) {
                Text(text = "INITIALIZE")
            }
            Button(
                onClick = {
                    viewModel.runInference()
                },
                modifier = Modifier.padding(innerPadding)
            ) {
                Text(viewModel.output + viewModel.elapsed)
            }
        }
    }
}



NnApiViewModel.kt
class NnApiViewModel : ViewModel() {

    var options = Interpreter.Options()
    var nnApiDelegate: NnApiDelegate? = null
    var tfLite: Interpreter? = null
    var output by mutableStateOf("")
    var elapsed by mutableStateOf(0L)

    // Initialize interpreter with NNAPI delegate for Android Pie or above
    fun initialize(applicationContext: Context) {
        nnApiDelegate = NnApiDelegate()
        options.useNNAPI = true
        NnApiDelegate.Options().useNnapiCpu = false
        options.addDelegate(nnApiDelegate)

        val model = applicationContext.assets.open("models/whisper-tiny.tflite")
        val file = model.readBytes()

        val fileName = "whisper-tiny.tflite"
        applicationContext.openFileOutput(fileName, Context.MODE_PRIVATE).use {
            it.write(file)
        }

        val modelFile = File(applicationContext.filesDir,"whisper-tiny.tflite")

        // Initialize TFLite interpreter
        try {
            tfLite = Interpreter(modelFile, options)
        } catch (e: Exception) {
            throw RuntimeException(e)
        }
    }

    fun runInference() {
        try {
            val outputShape = tfLite?.getOutputTensor(0)
            val input = TensorBuffer.createFixedSize(intArrayOf(1, 80, 3000), DataType.FLOAT32)
            val output = TensorBuffer.createFixedSize(outputShape?.shape(), DataType.FLOAT32)

            val start = System.currentTimeMillis()
            tfLite?.run(input.buffer, output.buffer)
            elapsed = System.currentTimeMillis() - start
        } catch (e: RuntimeException) {
            throw RuntimeException(e)
        }
    }

    fun unload() {
        tfLite?.close()
        nnApiDelegate?.close()
    }
}
```


### Relevant log output

```shell
I  Loaded native library: tensorflowlite_jni
I  Didn't load native library: tensorflowlite_jni_gms_client
I  Initialized TensorFlow Lite runtime.
I  DeviceManager::DeviceManager
I  findAvailableDevices
E  Error opening trace file: No such file or directory (2)
I  Found interface google-edgetpu (version = 2.0)
I  Found interface google-armnn (version = ArmNN)
I  Created TensorFlow Lite delegate for NNAPI.
E  FATAL EXCEPTION: main                                                                                                                                                                                                   java.lang.RuntimeException: java.lang.IllegalArgumentException: Internal error: Error applying delegate:
```
</details>