GPUv2 syntax error in shader generation in broadcast with different batch sizes
Hi @gustavla ,

Have you tried replicating this particular issue on other android devices or this happens only on galaxy s23? @sawantkumar I also tried it on a Google Pixel 7 and it also falls back to CPU. Hi @gustavla ,

I ran the code to create the tflite model but it gave me the below error:



---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[8], line 23
     21 # Convert the model.
     22 converter = tf.lite.TFLiteConverter.from_keras_model(model)
---> 23 tflite_model = converter.convert()
     25 # Save the model.
     26 with open('model.tflite', 'wb') as f:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\lite.py:1175, in _export_metrics.<locals>.wrapper(self, *args, **kwargs)
   1172 @functools.wraps(convert_func)
   1173 def wrapper(self, *args, **kwargs):
   1174   # pylint: disable=protected-access
-> 1175   return self._convert_and_export_metrics(convert_func, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\lite.py:1129, in TFLiteConverterBase._convert_and_export_metrics(self, convert_func, *args, **kwargs)
   1127 self._save_conversion_params_metric()
   1128 start_time = time.process_time()
-> 1129 result = convert_func(self, *args, **kwargs)
   1130 elapsed_time_ms = (time.process_time() - start_time) * 1000
   1131 if result:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\lite.py:1641, in TFLiteKerasModelConverterV2.convert(self)
   1637 if saved_model_convert_result:
   1638   return saved_model_convert_result
   1640 graph_def, input_tensors, output_tensors, frozen_func = (
-> 1641     self._freeze_keras_model()
   1642 )
   1644 graph_def = self._optimize_tf_model(
   1645     graph_def, input_tensors, output_tensors, frozen_func
   1646 )
   1648 return super(TFLiteKerasModelConverterV2, self).convert(
   1649     graph_def, input_tensors, output_tensors
   1650 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\convert_phase.py:215, in convert_phase.<locals>.actual_decorator.<locals>.wrapper(*args, **kwargs)
    213 except Exception as error:
    214   report_error_message(str(error))
--> 215   raise error from None

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\convert_phase.py:205, in convert_phase.<locals>.actual_decorator.<locals>.wrapper(*args, **kwargs)
    202 @functools.wraps(func)
    203 def wrapper(*args, **kwargs):
    204   try:
--> 205     return func(*args, **kwargs)
    206   except ConverterError as converter_error:
    207     if converter_error.errors:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\lite.py:1582, in TFLiteKerasModelConverterV2._freeze_keras_model(self)
   1573 # If the model's call is not a `tf.function`, then we need to first get its
   1574 # input signature from `model_input_signature` method. We can't directly
   1575 # call `trace_model_call` because otherwise the batch dimension is set
   1576 # to None.
   1577 # Once we have better support for dynamic shapes, we can remove this.
   1578 if not isinstance(self._keras_model.call, _def_function.Function):
   1579   # Pass `keep_original_batch_size=True` will ensure that we get an input
   1580   # signature including the batch dimension specified by the user.
   1581   # TODO(b/169898786): Use the Keras public API when TFLite moves out of TF
-> 1582   input_signature = _model_input_signature(
   1583       self._keras_model, keep_original_batch_size=True
   1584   )
   1586 # TODO(b/169898786): Use the Keras public API when TFLite moves out of TF
   1587 func = _trace_model_call(self._keras_model, input_signature)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\tensorflow\lite\python\tflite_keras_util.py:84, in model_input_signature(model, keep_original_batch_size)
     82   input_specs = input_specs[0][0]
     83 else:
---> 84   input_specs = model._get_save_spec(  # pylint: disable=protected-access
     85       dynamic_batch=not keep_original_batch_size)
     86   if input_specs is None:
     87     return None

AttributeError: 'Functional' object has no attribute '_get_save_spec'
 Just like in the other issue, I think the repro only works on tensorflow 2.15.

Here is the model: https://qaihub-public-issues.s3.us-west-2.amazonaws.com/tflite/66712_model.tflite Hi @gustavla ,

Can you please use the code that i provided for model conversion in this [issue](https://github.com/tensorflow/tensorflow/issues/66729) and let me know if it resolved this issue. @sawantkumar I do not have access to the colab that you posted in that issue. Hi @gustavla ,

Given you the access.
**System information**
- Samsung Galaxy S23 / Android 13 / Snapdragon® 8 Gen 2 | SM8550 
- TFLite 2.16.1 (also repros on many older versions)

**Standalone code to reproduce the issue** 

Compile the model that fails:
```
import tensorflow as tf
import keras

input0_shape = [2, 2, 1]
input1_shape = [1, 2, 2, 1]
output_shape = [1, 2, 2, 1]

tf_input0 = keras.Input(input0_shape[1:], batch_size=input0_shape[0])
tf_input1 = keras.Input(input1_shape[1:], batch_size=input1_shape[0])


class MyMatMul(keras.layers.Layer):
    def call(self, tf_input0, tf_input1):
        tf_output = tf_input0 * tf_input1
        return tf_output

tf_output = MyMatMul()(tf_input0, tf_input1)

model = keras.Model(inputs=[tf_input0, tf_input1], outputs=[tf_output])

# Convert the model.
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)
```

Execute on GPUv2 (for instance through benchmark_tool). It falls back to CPU and executes without a problem there.

**Any other info / logs**

Runtime log (executed through https://aihub.qualcomm.com/):

```
[30/Apr/2024:21:28:08 +05:30: profiler/info] Detected chipset 2807, made by 2000.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] Loading tflite model Models/model.tflite
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] Malloc VM size before: 16488.0 kB, allocated: 14040.6 kB, slack: 2447.4 kB.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] Current memory baseline range: 58204.6-60652.0 kB.
[30/Apr/2024:21:28:08 +05:30: profiler/debug] [job_id: jqp4zo91g] [model.tflite] Runtime metadata not found in Models/model.tflite/trt_metadata.json or Models/model.tflite/trt_metadata.pb
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] TF Lite version 2.16.1. Loading model from Models/model.tflite.
[30/Apr/2024:21:28:08 +05:30: profiler/debug] [job_id: jqp4zo91g] [model.tflite] Mapping resource file in Models/model.tflite
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] Loaded model. Minimum TF Lite version = .
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] No delegates specified; using compute unit=cpu_and_gpu.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] [tflite] Initialized TensorFlow Lite runtime.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] GPUV2 delegate requested. OpenCL detected.
[30/Apr/2024:21:28:08 +05:30: profiler/debug] [job_id: jqp4zo91g] [model.tflite] Enabling delegate cache in dir=/data/user/0/ai.tetra.tungsten/cache/1714492688301/ai.tetra.runtime/0.6.0/model.tflite_8945450969824422876_1714492678194/gpuv2.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] [tflite] Created TensorFlow Lite delegate for GPU.
[30/Apr/2024:21:28:08 +05:30: profiler/info] [job_id: jqp4zo91g] [model.tflite] [tflite] Replacing 1 out of 1 node(s) with delegate (TfLiteGpuDelegateV2) node, yielding 1 partitions for the whole graph.
[30/Apr/2024:21:28:08 +05:30: profiler/warning] [job_id: jqp4zo91g] [model.tflite] [tflite] File /data/user/0/ai.tetra.tungsten/cache/1714492688301/ai.tetra.runtime/0.6.0/model.tflite_8945450969824422876_1714492678194/gpuv2/gpuv2_11394406828873984768.bin couldn't be opened for reading: No such file or directory
[30/Apr/2024:21:28:08 +05:30: profiler/warning] [job_id: jqp4zo91g] [model.tflite] [tflite] Failed to build program executable - Build program failureBC-src-code:64:105: error: expected expression
  {half4 second_value = read_imageh(src_tensor_1_image2d, smp_zero, (int2)((((X)) * shared_int4_0.w + (())), (0)));
                                                                                                        ^
1 diagnostic(s) generated.
```

There are many input shapes that will trigger this (NG), but some execute just fine (OK):
```
# OK                                                  
# (5, 352, 352, 3), (352, 352, 3) -> (5, 352, 352, 3) 
# (2, 1, 1, 3), (1, 3) -> (2, 1, 1, 3)                
# (2, 1, 1, 3), (2, 3) -> (2, 1, 2, 3)                
# (2, 1, 1, 3), (3, 3) -> (2, 1, 3, 3)                
# (2, 2, 2, 1), (2, 1) -> (2, 2, 2, 1)                
# (2, 2, 1), (2, 1) -> (2, 2, 1)                      
# (3, 2), (2,) -> (3, 2)                              
# (3,) -> (1, 2, 2, 3) -> (1, 2, 2, 3)                
# (2,), (1, 2) -> (1, 2)                              
# (2, 2), (1, 2) -> (2, 2)                            
                                                      
# NG                                                  
# (1, 2, 2, 3), (3,) -> (1, 2, 2, 3)                  
# (1, 2, 2, 1), (2, 2, 1) -> (1, 2, 2, 1)             
# (2, 2, 1), (1, 2, 2, 1) -> (1, 2, 2, 1)             
# (2, 1), (1, 2, 1) -> (1, 2, 1)                      
# (1, 2, 2, 1), (2, 1) -> (1, 2, 2, 1)                
# (1, 2, 1), (2, 1) -> (1, 2, 1)                      
# (1, 2), (2,) -> (1, 2)     
```
