E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inEfficientDet-D2/model/stack_0/block_1/drop/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
Hi @aurelliafirsty,
In order to expedite the trouble-shooting process, we request you to share the colab link to replicate the issue reported here.   Thank you!	
 https://colab.research.google.com/drive/14Kczx0SrM_t3zvLagbqtesnfF0gieEtZ?authuser=2 I'm facing a different error while replicating the above code in TF v2.9.  Please find the gist [here](https://colab.sandbox.google.com/gist/synandi/1866e1f3216d56981e4fc1964bcfa597/efficientdet.ipynb). Thank you! This issue has been automatically marked as stale because it has no recent activity. It will be closed if no further activity occurs. Thank you.
 Closing as stale. Please reopen if you'd like to work on this further.
 Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59578">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/59578">No</a>
 I encountered the same issue a few weeks ago. It was solved by reducing the batch size. I had to go from a batch size of 16 to 4 with 10gb of vram. > I encountered the same issue a few weeks ago. It was solved by reducing the batch size. I had to go from a batch size of 16 to 4 with 10gb of vram.

I am facing the same error, but I could not solve it by reducing the batch size, even to 1. The curious thing is that this error occurs only on Linux, not on Windows.  Facing the same error on Colab, strangely enough with a lower batch size the error doesn't actually stop the training, so I'm managing with a batch size of 4 (8 takes up more the 16GB of VRAM in my case) I have the same issue here

tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel_5/dropout_10/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer

this is my code

```
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import *

# Set the input shape of the images (adjust based on the input image size)
input_shape = (128, 128, 3)  # Adjust based on the input image size

# Set the number of segmentation classes 
n_classes = 1  # Number of segmentation classes

# Define the model architecture
inputs = Input(shape=input_shape)  # Define the input layer with the specified input shape

# Encoder
conv1 = Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)  # First convolutional layer with 64 filters
conv1 = BatchNormalization()(conv1)  # Apply batch normalization to normalize the activations
conv1 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv1)  # Second convolutional layer with 64 filters
conv1 = BatchNormalization()(conv1)  # Apply batch normalization to normalize the activations
pool1 = MaxPooling2D((2, 2))(conv1)  # Max pooling layer with a pool size of (2, 2)

conv2 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool1)  # Convolutional layer with 128 filters
conv2 = BatchNormalization()(conv2)  # Apply batch normalization to normalize the activations
conv2 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv2)  # Convolutional layer with 128 filters
conv2 = BatchNormalization()(conv2)  # Apply batch normalization to normalize the activations
pool2 = MaxPooling2D((2, 2))(conv2)  # Max pooling layer with a pool size of (2, 2)

conv3 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool2)  # Convolutional layer with 256 filters
conv3 = BatchNormalization()(conv3)  # Apply batch normalization to normalize the activations
conv3 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv3)  # Convolutional layer with 256 filters
conv3 = BatchNormalization()(conv3)  # Apply batch normalization to normalize the activations
pool3 = MaxPooling2D((2, 2))(conv3)  # Max pooling layer with a pool size of (2, 2)

conv4 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool3)  # Convolutional layer with 512 filters
conv4 = BatchNormalization()(conv4)  # Apply batch normalization to normalize the activations
conv4 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv4)  # Convolutional layer with 512 filters
conv4 = BatchNormalization()(conv4)  # Apply batch normalization to normalize the activations
drop4 = Dropout(0.5)(conv4)  # Apply dropout regularization with a rate of 0.5
pool4 = MaxPooling2D((2, 2))(drop4)  # Max pooling layer with a pool size of (2, 2)

conv5 = Conv2D(1024, (3, 3), activation='relu', padding='same')(pool4)  # Convolutional layer with 1024 filters
conv5 = BatchNormalization()(conv5)  # Apply batch normalization to normalize the activations
conv5 = Conv2D(1024, (3, 3), activation='relu', padding='same')(conv5)  # Convolutional layer with 1024 filters
conv5 = BatchNormalization()(conv5)  # Apply batch normalization to normalize the activations
drop5 = Dropout(0.5)(conv5)  # Apply dropout regularization with a rate of 0.5

# Decoder
up6 = concatenate([UpSampling2D((2, 2))(drop5), conv4], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv6 = Conv2D(512, (3, 3), activation='relu', padding='same')(up6)  # Convolutional layer with 512 filters
conv6 = BatchNormalization()(conv6)  # Apply batch normalization to normalize the activations
conv6 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv6)  # Convolutional layer with 512 filters
conv6 = BatchNormalization()(conv6)  # Apply batch normalization to normalize the activations

up7 = concatenate([UpSampling2D((2, 2))(conv6), conv3], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv7 = Conv2D(256, (3, 3), activation='relu', padding='same')(up7)  # Convolutional layer with 256 filters
conv7 = BatchNormalization()(conv7)  # Apply batch normalization to normalize the activations
conv7 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv7)  # Convolutional layer with 256 filters
conv7 = BatchNormalization()(conv7)  # Apply batch normalization to normalize the activations

up8 = concatenate([UpSampling2D((2, 2))(conv7), conv2], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv8 = Conv2D(128, (3, 3), activation='relu', padding='same')(up8)  # Convolutional layer with 128 filters
conv8 = BatchNormalization()(conv8)  # Apply batch normalization to normalize the activations
conv8 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv8)  # Convolutional layer with 128 filters
conv8 = BatchNormalization()(conv8)  # Apply batch normalization to normalize the activations

up9 = concatenate([UpSampling2D((2, 2))(conv8), conv1], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same')(up9)  # Convolutional layer with 64 filters
conv9 = BatchNormalization()(conv9)  # Apply batch normalization to normalize the activations
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv9)  # Convolutional layer with 64 filters
conv9 = BatchNormalization()(conv9)  # Apply batch normalization to normalize the activations

outputs = Conv2D(n_classes, (1, 1), activation='softmax')(conv9)  # Convolutional layer for output

# Create the model
model = Model(inputs=inputs, outputs=outputs)

# Print the model summary
model.summary()


# Set the optimizer for the model
optimizer = tf.keras.optimizers.Adam(lr=1e-4)

# Compile the model with loss function and metrics
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

images_path = '/kaggle/input/coco-2014-dataset-for-yolov3/coco2014/images/val2014'
masks_path = '/kaggle/working/mask_val_2014'
batch_size = 8

val_generator = CustomDataGenerator(images_path, masks_path, batch_size)


# Fit the model with the training generator
train_steps =  len(os.listdir( "/kaggle/working/mask_train_2014/"))/batch_size
model.fit(train_generator,validation_data = val_generator, steps_per_epoch = train_steps , epochs=20)
```

Also to ensure that all the input and output have proper shape I run the code below

```
def print_preprocessed_image_shapes(model, generator):
    """
    Print the shapes of preprocessed images generated by the provided model and generator.

    Args:
        model (tf.keras.Model): The trained model.
        generator (CustomDataGenerator): Instance of the CustomDataGenerator class.
    """
    for i in range(len(generator)):
        # Get a batch of preprocessed images from the generator
        batch_images, batch_masks = generator[i]

        # Print the shapes of the preprocessed images
        for image in batch_images:
            print(f"Shape of preprocessed image: {image.shape}")
        for mask in batch_maskss:

             print(f"Shape of preprocessed image: {mask.shape}")
            
# Print the shapes of preprocessed images
print_preprocessed_image_shapes(model, val_generator)
```


As a result of this error, the model was unable to undergo the training process.





 Related to https://github.com/tensorflow/tensorflow/issues/61324 and https://github.com/tensorflow/tensorflow/issues/34499 > Facing the same error on Colab, strangely enough with a lower batch size the error doesn't actually stop the training, so I'm managing with a batch size of 4 (8 takes up more the 16GB of VRAM in my case)

For me it's the same. The training was done successfully even if the error showed up. Does it sound reasonable to continue with this error?  > > Facing the same error on Colab, strangely enough with a lower batch size the error doesn't actually stop the training, so I'm managing with a batch size of 4 (8 takes up more the 16GB of VRAM in my case)
> 
> For me it's the same. The training was done successfully even if the error showed up. Does it sound reasonable to continue with this error? 

I've had no problem with the model after training so I think you can continue
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

No

### Source

source

### Tensorflow Version

2.9.2

### Custom Code

Yes

### OS Platform and Distribution

_No response_

### Mobile device

_No response_

### Python version

3.8

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

```shell
I want to train my dataset with model zoo, which is the EfficientDet D2 model, but I have an error 

E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inEfficientDet-
D2/model/stack_0/block_1/drop/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```


### Standalone code to reproduce the issue

```shell
!python model_main_tf2.py --pipeline_config_path=/mydrive/EfficientDet/data/ssd_efficientdet_d2_768x768_coco17_tpu-8.config --model_dir=/mydrive/EfficientDet/training --alsologtostderr
```


### Relevant log output

```shell
WARNING:tensorflow:Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
W0206 13:32:47.285290 139961433057024 utils.py:82] Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
W0206 13:33:01.063754 139961433057024 utils.py:82] Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
W0206 13:33:12.740289 139961433057024 utils.py:82] Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
WARNING:tensorflow:Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
W0206 13:33:26.074406 139961433057024 utils.py:82] Gradients do not exist for variables ['stack_6/block_1/expand_bn/gamma:0', 'stack_6/block_1/expand_bn/beta:0', 'stack_6/block_1/depthwise_conv2d/depthwise_kernel:0', 'stack_6/block_1/depthwise_bn/gamma:0', 'stack_6/block_1/depthwise_bn/beta:0', 'stack_6/block_1/project_bn/gamma:0', 'stack_6/block_1/project_bn/beta:0', 'top_bn/gamma:0', 'top_bn/beta:0'] when minimizing the loss. If you're using `model.compile()`, did you forget to provide a `loss` argument?
2023-02-06 13:33:33.925763: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inEfficientDet-D2/model/stack_0/block_1/drop/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```
</details>