Slower inference since TensorFlow Lite 2.12
Hi @pkgoogle,
Could you please look into the issue
Thank you Hi @Qheb, I'm looking into this, do you know what Android SDK and API Level you are using? Hi @pkgoogle , thanks for looking into this.
I reproduced the issue while compiling & targeting the Android SDK 34. @Qheb, Thanks!, by API version I meant which version of Android are you using?

![image](https://github.com/tensorflow/tensorflow/assets/132095473/e854bbe4-4c04-4b62-9a0e-b66f54290f48)

Thanks for your help.
 Oh, sorry!

I tested it on two devices:
- Samsung Galaxy s23 Ultra (`samsung/dm3qxeea/dm3q:14/UP1A.231005.007/S918BXXS3BWK5:user/release-keys`) running Android 14 (API level 34)
- Samsung Galaxy A51 (`samsung/a51nseea/a51:13/TP1A.220624.014/A515FXXU8HWI1:user/release-keys`) running Android 13 (API level 33) Hi @Qheb, have you tried Google play services versions 2.12 through 2.14? I don't see it above. Additionally how are you making these measurements? Custom code to measure? A benchmarking tool of some sort? Something else in Android Studio? As I currently see this, it seems like there is no issue with version 2.15 which is the latest... as such while it does seem peculiar, I don't really see this as an issue worth investigating. Hi @pkgoogle,

Is there a way to choose which TFLite version I can use through the Google Play Services ? I seems to me that I can only use the one currently available on the device. The week before I opened this issue, it was `2.13.0`, right now it is `2.15.0`. And with both I have "normal" inference times.

About the measurements, I am using the measurements [logged into the example app](https://github.com/tensorflow/examples/blob/master/lite/examples/pose_estimation/android/app/src/main/java/org/tensorflow/lite/examples/poseestimation/ml/PoseNet.kt#L100):
```kotlin
val inferenceStartTimeNanos = SystemClock.elapsedRealtimeNanos()
interpreter.runForMultipleInputsOutputs(inputArray, outputMap)
lastInferenceTimeNanos = SystemClock.elapsedRealtimeNanos() - inferenceStartTimeNanos
Log.i(
    TAG,
    String.format("Interpreter took %.2f ms", 1.0f * lastInferenceTimeNanos / 1_000_000)
)
```

Were you able to test it on a bundled `2.15` version ? I can only have up to the `2.14` on [Maven Central](https://repo.maven.apache.org/maven2/org/tensorflow/tensorflow-lite/). But I tried the nightly build, which is a `2.16` and it still haves the issue:

TFLite version: `2.16.0-nightly` (bundled)
Average inference time: `24.09ms`

It seems to me that there is a difference between the Google Play Services builds, and the one from the bundled version. Is this possible ?

About Maven Central, is this normal that the `2.15.0` hasn't been published on it ?

Thank you for your time. Hi @Qheb, just to make sure we're talking about the same environment, how are you bundling 2.16.0-nightly? As you said, 2.15.0 is not currently published in Maven Central. Are you building an .aar and bundling that with your project? Like here? https://www.tensorflow.org/lite/android/lite_build#use_nightly_snapshots or are you doing something else? If that is performing poorly then we can look into it, as it's currently underperforming. Thanks for your help and any additional information you can provide. Hello @pkgoogle,
I am **not** building the nightly .aar myself.
Instead I use, the `ossrh-snapshot` Maven repository, like in the link you provided, and I set the tflite dependencies to:

```groovy
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly-SNAPSHOT'
implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly-SNAPSHOT'
``` Hi @Qheb, I was able to test on a Galaxy S23 Ultra, with the nightly snapshot:

```
2024-01-04 11:27:08.997 21876-22863 Posenet                 org....lite.examples.poseestimation  I  Interpreter took 36.65 ms
2024-01-04 11:27:09.051 21876-22863 Posenet                 org....lite.examples.poseestimation  I  Interpreter took 36.60 ms
2024-01-04 11:27:09.106 21876-22863 Posenet                 org....lite.examples.poseestimation  I  Interpreter took 36.49 ms
2024-01-04 11:27:09.161 21876-22863 Posenet                 org....lite.examples.poseestimation  I  Interpreter took 36.39 ms
2024-01-04 11:27:09.215 21876-22863 Posenet                 org....lite.examples.poseestimation  I  Interpreter took 36.82 ms
```

To use Google Play Services, it seems I have to change some code in the demo, do you have the changes you made to use Google Play Services? so that I can replicate your environment exactly? (or did you change to GPS some other way?) Thanks for your help. Hi @pkgoogle, here are the two commits I used for running the Google Play Services:

1. [1-update-everything.txt](https://github.com/tensorflow/tensorflow/files/13841289/1-update-everything.txt): Blindly update dependencies so I can run the project (I suppose you've already done it on your side)
2. [2-TfLite-gps.txt](https://github.com/tensorflow/tensorflow/files/13841290/2-TfLite-gps.txt): GMS dependency + Initialize TFLite before creating the pose estimator + set TFLite runtime to FROM_SYSTEM_ONLY

 Thanks @Qheb, I was able to apply your patches directly and am seeing the same performance improvement:

```
2024-01-05 14:12:58.689  2724-3081  Posenet                 org....lite.examples.poseestimation  I  Interpreter took 14.93 ms
2024-01-05 14:12:58.722  2724-3081  Posenet                 org....lite.examples.poseestimation  I  Interpreter took 12.08 ms
2024-01-05 14:12:58.763  2724-3081  Posenet                 org....lite.examples.poseestimation  I  Interpreter took 14.98 ms
2024-01-05 14:12:58.796  2724-3081  Posenet                 org....lite.examples.poseestimation  I  Interpreter took 13.62 ms
2024-01-05 14:12:58.820  2724-3081  Posenet                 org....lite.examples.poseestimation  I  Interpreter took 11.32 ms
```

At least 2, sometimes 3x the speed on the same device,

@arfaian, can you please take a look? Thanks. Hello, sorry to insist, but is there any news about this ?
**System information**
- Android Device information: `samsung/dm3qxeea/dm3q:14/UP1A.231005.007/S918BXXS3BWK5:user/release-keys` (reproduced on any device I tested)
- Bundled TFLite version from [Maven](https://repo.maven.apache.org/maven2/org/tensorflow/tensorflow-lite/): `2.12.0` and later
- TensorFlow Lite in Play Services SDK version: `16.1.0`
- Google Play Services version: `23.45.23 (190400-587848529)`

**Standalone code to reproduce the issue**
I used the [TensorFlow Lite Pose Estimation Android Demo](https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/android) where I changed the TensorFlow Lite version

**Any other info / logs**
With any bundled version of TFLite >= 2.12.0, the inference time of some model is twice the inference time with TFLite v2.11.0. This issue is **not** reproduced with the TFLite from the Play Services.

Especially when I compared last week the bundled 2.13.0 against the 2.13.0 from the Play Services. The bundled version inference times were twice the ones of the Play Services.
(I can't test it anymore since the version from the Play Services has been updated to 2.15.0 and this one is not available on Maven)

For example, on a Samsung Galaxy S23 ultra, the PoseNet model from the example app:

TFLite version: `2.11.0 (bundled)`
Average inference time: `12.28ms`

TFLite version: `2.12.0 (bundled)`
Average inference time: `25.93ms`

TFLite version: `2.13.0 (bundled)`
Average inference time: `26.03ms`

TFLite version: `2.14.0 (bundled)`
Average inference time: `25.86ms`

TFLite version: `2.15.0 (Google Play Services)`
Average inference time: `12.12ms`

Any idea where that could come from ?