TFLite: On device Training Fails. ERROR: Node number 69 (FlexReluGrad) failed to prepare.
Hi @Bhuvan-1 

To use the `select_ops` we require to build the flex delegate using bazel as mentioned and also recommended.

I have tried it in Mac M1 and I was successfully able to be build the flex delegate. Please find the screenshot.

<img width="567" alt="image" src="https://github.com/tensorflow/tensorflow/assets/118897289/b720db43-7cce-4d52-8b4e-026e415c5340">


Can you try [this](https://github.com/tensorflow/tensorflow/issues/55536#issuecomment-1262358520) steps for CMake and see if it works for you? @pjpratik , The issue in my pc is that, when I try to build the library it runs for some time, and the terminal gets killed suddenly, probably due to Out Of Memory killer. So problem is with my pc ig. Where can I find a pre-built library? Hi @Bhuvan-1

The pre built for binaries for different tools are available [here](https://www.tensorflow.org/lite/performance/implementing_delegate#resources) but for flex delegate we may have to build on our own or try the 3rd party pre built binaries. 

Check [this](https://github.com/second-state/WasmEdge-tensorflow-deps/releases/download/TF-2.12.0-CC/WasmEdge-tensorflow-deps-TFLite-TF-2.12.0-CC-manylinux2014_aarch64.tar.gz) prebuilt library and see if it works for your case.

Thanks. It gives me this error while linking

I copied the library to /usr/local/lib  and the command used to compile is 

**g++ example.cpp -ltensorflowlite_flex**

/usr/bin/ld: skipping incompatible /usr/local/lib/libtensorflowlite_flex.so when searching for -ltensorflowlite_flex
/usr/bin/ld: cannot find -ltensorflowlite_flex: No such file or directory
/usr/bin/ld: skipping incompatible /usr/local/lib/libtensorflowlite_flex.so when searching for -ltensorflowlite_flex
collect2: error: ld returned 1 exit status
` Hi @Bhuvan-1 

We need to link the `tensorflowlite_flex` in CMakeLists.txt and update the path to the `.so` file
Sample code snippet
```
find_library(TF_LIB tensorflowlite HINTS "${TENSORFLOW_SOURCE_DIR}/bazel-bin/tensorflow/lite/")
find_library(TF_LIB_FLEX tensorflowlite_flex HINTS "${TENSORFLOW_SOURCE_DIR}/bazel-bin/tensorflow/lite/delegates/flex/")
target_link_libraries(testmodel
-Wl,--no-as-needed # Need --no-as-needed to link tensorflowlite_flex
${TF_LIB}
${TF_LIB_FLEX}
)
```
Thanks.





 I tried the above way, and got this


usr/bin/ld: /mnt/c/Users/G BHUVAN REDDY/Desktop/TFL/LIBS/libtensorflowlite_flex.so: error adding symbols: file in wrong format
collect2: error: ld returned 1 exit status
make[2]: *** [CMakeFiles/demo.dir/build.make:166: demo] Error 1
make[1]: *** [CMakeFiles/Makefile2:1389: CMakeFiles/demo.dir/all] Error 2
make: *** [Makefile:130: all] Error 2 Hi @Bhuvan-1, it looks like you are using WSL or WSL 2. Since it seems like your .so file is in the Windows File system and not the linux one. Generally unless something specifically tells you to you should never cross file system boundaries (In this case it seems cmake is trying to read in the mounted windows directory. Can you put the .so file in /usr/lib or where ever your other .so files are? Update your CMakeLists.txt to point to location as well. It might be helpful to upload your CMakeLists.txt file as well.

Additionally please always let us know if you are using WSL/WSL 2 in the issue description as well. Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. @Bhuvan-1 have you worked on model merger? i mean after on device you are saving model.ckpt. how you merging tflite model with model.ckpt for inference?  @AIML-ankit , model.ckpt  only has the weights stored. So to use them, we need to use the `restore` function to restore the weights from model.ckpt into memory . Then we can run inference using the updated weights.
**System information**
- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu, 22.04
- TensorFlow installed from (source or binary): source
- TensorFlow version (or github SHA if from source): r2.12


**Provide the text output from tflite_convert**

I used [On device training](https://www.tensorflow.org/lite/examples/on_device_training/overview) guide to convert to tflite model and then stored it as model.tflite


Then I used this c++ code to invoke the "train" signature Runner and it fails.


**Standalone code to reproduce the issue** 
```

    std::unique_ptr<FlatBufferModel> model = FlatBufferModel::BuildFromFile("../model.tflite");
    tflite::ops::builtin::BuiltinOpResolver builtin_resolver; 
    std::unique_ptr<Interpreter> interpreter;
    if( InterpreterBuilder(*model, builtin_resolver)(&interpreter) != kTfLiteOk){
        printf("Failed to build interpreter\n");
        exit(0);
    }

    // RESIZE Input Tensor Shape, if needed, then call ->AllocateTensors()
    if( interpreter->AllocateTensors() != kTfLiteOk){
        printf("Failed to allocate tensors!\n");
        exit(0);
    }

    //train model


    SignatureRunner* trainSignatureRunner = interpreter->GetSignatureRunner( "train" );
    trainSignatureRunner->AllocateTensors();

    TfLiteTensor* x_input_tensor = trainSignatureRunner->input_tensor( "x" );
    TfLiteTensor* y_input_tensor = trainSignatureRunner->input_tensor( "y" );


    // fill x_input_tensor
    for(int i = 0; i < 28*28; i++){
        x_input_tensor->data.f[i] = 0.5;
    }

    // fill y_input_tensor
    for(int i = 0; i < 10; i++){
        y_input_tensor->data.f[i] = 0.0;
    }
    y_input_tensor->data.f[3] = 1.0;


    int EPOCHS = 5;
    for(int epoch = 0; epoch < EPOCHS; epoch++){

        trainSignatureRunner->Invoke();

        const TfLiteTensor* loss_output = trainSignatureRunner->output_tensor( "loss" );

        printf("Epoch %d, Loss : %.4f\n", epoch, loss_output->data.f[0]);
    }



```

I get this output  , after compiling the file using CMakeLists.txt inside the /lite/examples/minimal/ directory and running the executable

```
ERROR: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_select
ERROR: Node number 69 (FlexReluGrad) failed to prepare.
Epoch 0, Loss : 0.7413
ERROR: Select TensorFlow op(s), included in the given model, is(are) not supported by this interpreter. Make sure you apply/link the Flex delegate before inference. For the Android, it can be resolved by adding "org.tensorflow:tensorflow-lite-select-tf-ops" dependency. See instructions: https://www.tensorflow.org/lite/guide/ops_select
ERROR: Node number 69 (FlexReluGrad) failed to prepare.
Epoch 1, Loss : 0.7413

..... same for all epochs
```


I did refer to https://www.tensorflow.org/lite/guide/ops_select and I tried building the flex delegate library using 

```
bazel build -c opt --config=monolithic tensorflow/lite/delegates/flex:tensorflowlite_flex
```
This runs for a long time, and suddenly the terminal gets killed after finishing like 70%. 


Can someone suggest a way to include the flex delegate library using some hacks in CMakeLists.txt or any other alterative way
