TFLite inference order is not the same as TensorFlow model
Hi, i am new around here, and I was hoping for some guidance on how I can contribute to tensorflow specifically and open source in general. 

Is there something i should go through first before giving a shot at this issue? Hi, I am also new to TensorFlow lite. You can try this out with a (1, 7, 7, 3) tensor input and will have a (1, 3, 3, 1) output feature map as a result. 

```
custom_layer = PatchBasedConv2D()
input_tensor = tf.keras.layers.Input(shape = input_tensor.shape)
output_tensor = custom_layer(input)
model = tf.keras.Model(inputs=input_tensor, outputs=output_tensor)
```
This would work fine, but my problem is that when I look at the converted tflite model using html, it seems that the inference order has changed. In my custom layer, the inference should be done patch by patch, however, the tflite model expands all the patch first and inference by layer-by-layer. Hi @keltonlee, there are a lot of things that happen on conversion such that the architecture does not necessarily stay exactly the same. If you are familiar with how high languages get translated to machine code, you can think of it as how the same program can be translated to a different set of machine instructions. So, this is not necessarily an issue ... unless do you see performance degradation? (either in speed, accuracy, some other ML measure?), is the output extremely different or possibly degenerate for the same input? To give you an idea of you can see the passes that get applied on conversion: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/compiler/mlir/lite/tf_tfl_passes.h and trace through the logic.

Hi @Saurabh-Mokashi, I appreciate your enthusiasm. First review: https://github.com/tensorflow/tensorflow/blob/master/CONTRIBUTING.md

Unfortunately, I do not know your background so it's hard to say what you should do next, for Tensorflow specifically, I recommend a MOOC or course that teaches TF and related concepts in ML and AI. Create your own custom TF model if you haven't and try to train and evaluate it, perhaps create your own service if you have the skills. For actually adjusting and contributing to the framework, you should start with understanding Python and C++ generally. For most frameworks, Templated Meta Programming in C++ is usually a requirement, we moved to C++17 so be sure to be up to date w/ this version of C++. If you are confident in all that then pick an issue and start digging (you will likely need a debugging tool, logging, or print statements), you will probably get stuck, that means you have something to learn, awesome, go learn it and keep going until you are able to resolve an issue.

For TFLite, understand that TF represents everything as a computation graph, and that every computer program can be reduced to a computational graph as well (every computer program can be reduced to a series of machine instructions, which is a very simple graph with nodes in a series -- one input, one output). Thus you can use the same abstractions as compilers to convert/change this computational graph. If you haven't taken a compilers class, this is recommended. Not all issues require all this understanding but if you understand all of the above, contributing will be easier. This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61891">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61891">No</a>

### 1. System information

- OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS
- TensorFlow installation (pip package or built from source): pip
- TensorFlow library (version, if pip package or github SHA, if built from source): 2.13.0

### 2. Code

This is a custom depth-first inference layer, I try it using a (1, 7, 7, 4) tensor, the output is the same as the two layer cnn model and the output size is (1, 3, 3, 1).

```
create_conv_layer1 = Conv2D(filters=4, kernel_size=(3, 3), activation='relu', trainable=False, kernel_initializer=tf.initializers.Constant(0.5))
create_conv_layer2 = Conv2D(filters=1, kernel_size=(3, 3), activation='relu', trainable=False, kernel_initializer=tf.initializers.Constant(0.5))
```

```
class PatchBasedConv2D(Layer):
    def __init__(self, **kwargs):
        super(PatchBasedConv2D, self).__init__(**kwargs)
        self.layer_number = 2
        self.output_size = (3, 3, 1)
        self.expand_size = (1, 1)
        self.patch_stride = None
    
    @staticmethod
    def compute_last_patch_size(output_size, kernel_size, stride):
        input_height = (output_size[0] - 1) * stride[0] + kernel_size[0]
        input_width = (output_size[1] - 1) * stride[1] + kernel_size[1]
        return (input_height, input_width)
    
    def calculate_patch_count(self, input_size, patch_size, stride):
        return ((input_size[0] - patch_size[0]) // stride[0] + 1) * ((input_size[1] - patch_size[1]) // stride[1] + 1)
    
    def get_current_patch_possition(self, input_size, patch_size, stride, current_round):
        return ((current_round // ((input_size[1] - patch_size[1]) // stride[1] + 1)) * stride[0],
              (current_round % ((input_size[1] - patch_size[1]) // stride[1] + 1)) * stride[1])

    def build(self, input_shape):
        with tf.device('/CPU:0'):
            self.conv1 = create_conv_layer1
            self.conv2 = create_conv_layer2
            self.patch_size_tmp = self.compute_last_patch_size(self.expand_size, self.conv2.kernel_size, self.conv2.strides)
            self.patch_size = self.compute_last_patch_size(self.patch_size_tmp, self.conv1.kernel_size, self.conv1.strides)
            self.patch_stride = self.conv1.strides

    def call(self, inputs):
        if self.conv1.padding == 'same':
            inputs = tf.pad(inputs, [[0, 0], [1, 1], [1, 1], [0, 0]], mode='CONSTANT')
            
        num_patch = self.calculate_patch_count((inputs.shape[1], inputs.shape[2]), self.patch_size, self.patch_stride)
        
        number_patch_in_row = int(self.output_size[1] // self.expand_size[1])
        output_feature_map_tmp = None 
        output_feature_map = None
        
        for current_round in range(num_patch):
            position = get_current_patch_possition((inputs.shape[1], inputs.shape[2]), self.patch_size, self.patch_stride, current_round)
            patch_output = self.conv1(inputs[:, position[0]: position[0] + self.patch_size[0], position[1]: position[1] + self.patch_size[1], :])
            patch_output = self.conv2(patch_output)
            
            if output_feature_map_tmp is None:
                output_feature_map_tmp = patch_output
            else:
                output_feature_map_tmp = tf.concat([output_feature_map_tmp, patch_output], axis=2)
            
            if (current_round + 1) % number_patch_in_row == 0 and current_round != 0:
                if output_feature_map is None:
                    output_feature_map = output_feature_map_tmp
                else:
                    output_feature_map = tf.concat([output_feature_map, output_feature_map_tmp], axis=1)
                output_feature_map_tmp = None
        
        return output_feature_map
```

### 3. Failure after conversion
I am working on implementing depth-first inference on Tflite micro, above is my custom layer. You can see that in the call function, I picked out a patch of the input image and did 2 layers of Conv first, then went on to the next patch. It works just fine in my jupyter-notebook. However, when I convert it to tflite using TFLiteConverter, I found that it first expand all the patches, then do 2 Conv on all the patches. This turns out to be an inference layer-by-layer and uses up even more memory, why is that?


![截圖 2023-09-18 下午7 33 19](https://github.com/tensorflow/tensorflow/assets/68526411/e3768de1-b03f-44a8-8a61-51318c8b8f92)