Running quantized tflite model in "full" tensorflow
Hi @luikaire,

Could you please confirm that your code used delegates?. If not, use gpu delegates to accelerate the inference. Refer the[ link](https://www.tensorflow.org/lite/performance/gpu) for reference.

Thank You
 I use the tensorflow-rocm pip package downloaded from the official installation page (tensorflow-2.13.0.570+rocm-5.7.2) on a linux machine (ubuntu-22.04)

with this installation gpu support in "full" tensorflow/keras models works out of the box

but when running the tflite model file through tf.lite.Interpreter , only cpu is used. This means the tflite file, despite being a lot smaller (23 Mb) vs the keras .h5 model (91 Mb) runs slower

Are gpu delegates available for this setup (ubuntu linux+amd rocm)? If so, how can I force tflite to use gpu delegates?

The link you referred doesn't seem to have information about linux machines..


An earlier issue (https://github.com/tensorflow/tensorflow/issues/44129#issuecomment-712487197) mentions that GPU delegates are only available for android/ios
Is that still the case with the latest tensorflow as well? Hi @luikaire,

The link provided in the earlier comment exactly refers to gpu delegates for android and ios only.  In your usecase, the gpu deligates  may be build on linux through Bazel. Could you please try to follow these steps :
1. Install  Bazel:
    ```wget https://github.com/bazelbuild/bazel/releases/download/6.1.0/bazel-6.1.0-installer-linux-x86_64.sh```
2. Clone Tensorflow Repository
     ``` git clone https://github.com/tensorflow/tensorflow.git tensorflow_src```
3.  ```apt get install mesa-common-dev libgl1-mesa-dev libgles2-mesa-dev ocl-icd-opencl-dev```
4.  Run ./configure
5.  Run ```bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 
     tensorflow/lite/delegates/gpu:gl_delegate```.

https://github.com/tensorflow/tensorflow/issues/46498, https://github.com/tensorflow/tensorflow/issues/46498

Thank You


    


 This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62815">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/62815">No</a>

I quantized a full tensorflow model and got a smaller tflite file

This runs well using tf.lite.Interpreter. However, on my machine, tflite can only use CPU, while "full" tensorflow is much faster as it can use GPU as well 
Because of CPU-only execution, the smaller quantized model runs slower in tf.lite.Interpreter than the larger/original model on "full" tensorflow

So I would like to know if there's any way to run the quantized tflite model (which is saved as a .tflite file on disk) in "full" tensorflow (via load_model for example) so that GPU acceleration can be used

thanks