tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel_6/dropout_12/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
Hi @Armanasq,

In order to expedite the trouble-shooting process, please provide the complete code snippet to reproduce the issue reported here. I've tried to reproduce the issue [here](https://colab.research.google.com/gist/Varsha-anjanappa/55f59c3611c314f93550473980130af8/61324.ipynb), could you please take a look and let me know if i'm missing something . 

Please find a similar issue [here](https://github.com/tensorflow/tensorflow/issues/34499) and let us know if it helps.

Thank you!! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Same issue Hi @YanSte ,

Please share colab link or simple standalone code with supporting files to reproduce the issue in our environment.It helps us in localizing the issue faster.

Thanks! Here are the projects:
[Rice Classification](https://www.kaggle.com/code/yannicksteph/cnn-cv-rice-classification )
[MNIST Classification](https://www.kaggle.com/code/yannicksteph/cnn-cv-mnist-classification)

Always at the first epoch, I have:
`2023-08-09 13:10:29.937931: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential_2/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer`

It seems that adding a Dropout after a MaxPooling2D, Conv2D creates this problem.

`TF version 1.7.0-rc1`

Do you need more information? I got the same issue. 
[One notebook](https://www.kaggle.com/code/tsrdjan/second-try#Train) shows the issue, while [the other](https://www.kaggle.com/code/tsrdjan/kaggle-xy-h5-drugi-pokusaj#6.-Build-Deep-Learning-Model) does not.

Both of them are transfer learning. What is interesting to me is that I got this issue only when I try to use EfficientNets. At first, it was working fine (like it did in the second notebook, `kaggle-xy.h5-drugi-pokusaj`), but my newer attempts always have this issue. I tried to replicate architecture from 2nd notebook but it still doesn't remove the issue. So, as far as I can tell, this maybe has something to do with the input and processing part.

The 2nd notebook does have `Dropout` after pre-trained part, `GlobalAveragePooling2D` and `BatchNormalization` (in this order) and it was not an issue in my case. > I got the same issue. [One notebook](https://www.kaggle.com/code/tsrdjan/second-try#Train) shows the issue, while [the other](https://www.kaggle.com/code/tsrdjan/kaggle-xy-h5-drugi-pokusaj#6.-Build-Deep-Learning-Model) does not.
> 
> Both of them are transfer learning. What is interesting to me is that I got this issue only when I try to use EfficientNets. At first, it was working fine (like it did in the second notebook, `kaggle-xy.h5-drugi-pokusaj`), but my newer attempts always have this issue. I tried to replicate architecture from 2nd notebook but it still doesn't remove the issue. So, as far as I can tell, this maybe has something to do with the input and processing part.
> 
> The 2nd notebook does have `Dropout` after pre-trained part, `GlobalAveragePooling2D` and `BatchNormalization` (in this order) and it was not an issue in my case.

UPDATE: I managed to fix the issue by recreating the architecture and adding `training=False`.

**First version**:
```Python
import tensorflow as tf
from tensorflow.keras.applications.efficientnet_v2 import EfficientNetV2M
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, BatchNormalization, Dropout
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import regularizers

base_model = EfficientNetV2M(weights='imagenet', include_top=False, input_shape=(380, 380, 3), include_preprocessing=True)

base_model.trainable = False

top_dropout_rate = 0.2
x = base_model.output
x = GlobalAveragePooling2D()(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = BatchNormalization()(x)
x = Dropout(top_dropout_rate, name="top_dropout")(x)
x = Dense(1, activation='sigmoid', name="prediction")(x)

model = tf.keras.Model(inputs=base_model.input, outputs=x)
```

**Second version**:
```Python
from tensorflow.keras.applications.efficientnet_v2 import EfficientNetV2M, preprocess_input
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Dense, GlobalAveragePooling2D, BatchNormalization, Dropout, Input
from tensorflow.keras.optimizers import Adam
from tensorflow.keras import regularizers

base_model = EfficientNetV2M(weights='imagenet', include_top=False, input_shape=input_shape, include_preprocessing=True)
base_model.trainable = False

top_dropout_rate = 0.2
input_tensor = Input(shape=input_shape)
preprocessed_input = preprocess_input(input_tensor)

x = base_model(preprocessed_input, training=False)
x = GlobalAveragePooling2D()(x)
x = Dense(256, activation='relu', kernel_regularizer=regularizers.l2(0.05))(x)
x = BatchNormalization()(x)
x = Dropout(top_dropout_rate, name="top_dropout")(x)
x = Dense(1, activation='sigmoid', name="prediction")(x)

model = Model(inputs=input_tensor, outputs=x)
```

The final version similar to 2nd, but with `x = base_model(preprocessed_input)` changed to `x = base_model(preprocessed_input, training=False)`. Adding this fixed the issue in second version.

The information for these changes was found in [keras's documentation for transfer learning](https://keras.io/guides/transfer_learning/). This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. Confirmed the error even after the modification by @TodorovicSrdjan (similar to version 2, using MobileNet V2 for transfer learning). Here the code:

```python
 # Create the base model from the pre-trained MobileNet V2
 base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
                                               include_top=False,
                                               weights='imagenet')
 base_model.trainable = False
 data_augmentation = tf.keras.Sequential(
     [
         tf.keras.layers.RandomFlip("horizontal"),
         tf.keras.layers.RandomRotation(0.2),
     ]
 )

 inputs = tf.keras.Input(shape=IMG_SHAPE)

 x = data_augmentation(inputs)  # Apply random data augmentation
 x = tf.keras.applications.mobilenet_v2.preprocess_input(x)
 x = base_model(x, training=False)
 x = tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu')(x)
 x = tf.keras.layers.Dropout(0.2)(x)
 x = tf.keras.layers.GlobalAveragePooling2D()(x)
 x = tf.keras.layers.Dense(units=N_CLASSES, activation='softmax')(x)
 model = tf.keras.models.Model(inputs=inputs, outputs=x)
``` 

And the output:

```
Model: "model"
_________________________________________________________________
 Layer (type)                Output Shape              Param #
=================================================================
 input_2 (InputLayer)        [(None, 160, 160, 3)]     0

 sequential (Sequential)     (None, 160, 160, 3)       0

 mobilenetv2_1.00_160 (Func  (None, 5, 5, 1280)        2257984
 tional)

 conv2d (Conv2D)             (None, 3, 3, 32)          368672

 dropout (Dropout)           (None, 3, 3, 32)          0

 global_average_pooling2d (  (None, 32)                0
 GlobalAveragePooling2D)

 dense (Dense)               (None, 7)                 231

=================================================================
Total params: 2626887 (10.02 MB)
Trainable params: 368903 (1.41 MB)
Non-trainable params: 2257984 (8.61 MB)
_________________________________________________________________
Number of trainable weights = 4
Epoch 1/3
2023-08-21 12:39:38.540515: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```

TF 2.13.0 on cuDNN 8.6.0 and CUDA 12.2 on Linux x86_64.

**Update: same code does not generate any error on MacBookPro M1 Pro with TF 2.13.0 and Metal backend** @Varsha-anjanappa This bug appears in many open issues, including [61687](https://github.com/tensorflow/tensorflow/issues/61687), has there been any progress on fixing the issue? @meekus-fischer I think it is related to adding dropout after the 2D CNN. after removing them the error has been fixed. However I am not sure. I decided to use PyTorch. Same issue here on TF 2.14 with a custom model, with layers like below as others have mentioned previously:

```
tf.keras.layers.Conv2D(64, 3, activation='relu'),
tf.keras.layers.MaxPooling2D(),
tf.keras.layers.Dropout(0.4),
```

 I have similar/same error: "2023-11-03 17:20:43.612803: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:961] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential_10/dropout_31/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer"

I can confirm it goes away when I comment out the dropout after the MaxPooling2D layers.

```
		model.add(Conv2D(64, (3, 3), padding="same"))
		model.add(Activation("relu"))
		model.add(BatchNormalization(axis=chanDim))
		model.add(Conv2D(64, (3, 3), padding="same"))
		model.add(Activation("relu"))
		model.add(BatchNormalization(axis=chanDim))
		model.add(MaxPooling2D(pool_size=(2, 2)))
		model.add(Dropout(0.25)) <--- commenting out this line and the error goes away
``` See edumotya's response in #34499, as he suggests disabling the optimizer. This appears to have worked around the issue for me, although there's still something to be understood or resolved in the optimizer and/or Dropout's channel-first vs channel-last approach when the optimizer is enabled. > Confirmed the error even after the modification by @TodorovicSrdjan (similar to version 2, using MobileNet V2 for transfer learning). Here the code:
> 
> ```python
>  # Create the base model from the pre-trained MobileNet V2
>  base_model = tf.keras.applications.MobileNetV2(input_shape=IMG_SHAPE,
>                                                include_top=False,
>                                                weights='imagenet')
>  base_model.trainable = False
>  data_augmentation = tf.keras.Sequential(
>      [
>          tf.keras.layers.RandomFlip("horizontal"),
>          tf.keras.layers.RandomRotation(0.2),
>      ]
>  )
> 
>  inputs = tf.keras.Input(shape=IMG_SHAPE)
> 
>  x = data_augmentation(inputs)  # Apply random data augmentation
>  x = tf.keras.applications.mobilenet_v2.preprocess_input(x)
>  x = base_model(x, training=False)
>  x = tf.keras.layers.Conv2D(filters=32, kernel_size=3, activation='relu')(x)
>  x = tf.keras.layers.Dropout(0.2)(x)
>  x = tf.keras.layers.GlobalAveragePooling2D()(x)
>  x = tf.keras.layers.Dense(units=N_CLASSES, activation='softmax')(x)
>  model = tf.keras.models.Model(inputs=inputs, outputs=x)
> ```
> 
> And the output:
> 
> ```
> Model: "model"
> _________________________________________________________________
>  Layer (type)                Output Shape              Param #
> =================================================================
>  input_2 (InputLayer)        [(None, 160, 160, 3)]     0
> 
>  sequential (Sequential)     (None, 160, 160, 3)       0
> 
>  mobilenetv2_1.00_160 (Func  (None, 5, 5, 1280)        2257984
>  tional)
> 
>  conv2d (Conv2D)             (None, 3, 3, 32)          368672
> 
>  dropout (Dropout)           (None, 3, 3, 32)          0
> 
>  global_average_pooling2d (  (None, 32)                0
>  GlobalAveragePooling2D)
> 
>  dense (Dense)               (None, 7)                 231
> 
> =================================================================
> Total params: 2626887 (10.02 MB)
> Trainable params: 368903 (1.41 MB)
> Non-trainable params: 2257984 (8.61 MB)
> _________________________________________________________________
> Number of trainable weights = 4
> Epoch 1/3
> 2023-08-21 12:39:38.540515: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
> ```
> 
> TF 2.13.0 on cuDNN 8.6.0 and CUDA 12.2 on Linux x86_64.
> 
> **Update: same code does not generate any error on MacBookPro M1 Pro with TF 2.13.0 and Metal backend**

Hi @zzambia , I have tested your code and it seems working fine on colab though. Please check the [gist](https://colab.research.google.com/gist/SuryanarayanaY/7d76bd7471d3f17da6dedc5785d9c533/61324.ipynb) and confirm .

Hi @Armanasq , Could you please confirm whether this is still an issue. If so please submit a minimal code snippet to reproduce the issue. 

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61324">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61324">No</a>
 using `tf.nn.dropout(x, 0.5)` throws `E tensorflow/core/grappler/optimizers/meta_optimizer.cc:961] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inencoder_decoder_1/decoder_1/decoder_block_1/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer`

However, replacing it with `tf.keras.layers.Dropout(0.5)` resolved the issue for me. 

TF version: 2.17.0
### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

2.12.0

### Custom code

Yes

### OS platform and distribution

Kaggle

### Mobile device

_No response_

### Python version

3.10.12

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I am trying to train model below

```
import tensorflow as tf
from tensorflow.keras.models import Model
from tensorflow.keras.layers import *

# Set the input shape of the images (adjust based on the input image size)
input_shape = (128, 128, 3)  # Adjust based on the input image size

# Set the number of segmentation classes 
n_classes = 1  # Number of segmentation classes

# Define the model architecture
inputs = Input(shape=input_shape)  # Define the input layer with the specified input shape

# Encoder
conv1 = Conv2D(64, (3, 3), activation='relu', padding='same')(inputs)  # First convolutional layer with 64 filters
conv1 = BatchNormalization()(conv1)  # Apply batch normalization to normalize the activations
conv1 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv1)  # Second convolutional layer with 64 filters
conv1 = BatchNormalization()(conv1)  # Apply batch normalization to normalize the activations
pool1 = MaxPooling2D((2, 2))(conv1)  # Max pooling layer with a pool size of (2, 2)

conv2 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool1)  # Convolutional layer with 128 filters
conv2 = BatchNormalization()(conv2)  # Apply batch normalization to normalize the activations
conv2 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv2)  # Convolutional layer with 128 filters
conv2 = BatchNormalization()(conv2)  # Apply batch normalization to normalize the activations
pool2 = MaxPooling2D((2, 2))(conv2)  # Max pooling layer with a pool size of (2, 2)

conv3 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool2)  # Convolutional layer with 256 filters
conv3 = BatchNormalization()(conv3)  # Apply batch normalization to normalize the activations
conv3 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv3)  # Convolutional layer with 256 filters
conv3 = BatchNormalization()(conv3)  # Apply batch normalization to normalize the activations
pool3 = MaxPooling2D((2, 2))(conv3)  # Max pooling layer with a pool size of (2, 2)

conv4 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool3)  # Convolutional layer with 512 filters
conv4 = BatchNormalization()(conv4)  # Apply batch normalization to normalize the activations
conv4 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv4)  # Convolutional layer with 512 filters
conv4 = BatchNormalization()(conv4)  # Apply batch normalization to normalize the activations
drop4 = Dropout(0.5)(conv4)  # Apply dropout regularization with a rate of 0.5
pool4 = MaxPooling2D((2, 2))(drop4)  # Max pooling layer with a pool size of (2, 2)

conv5 = Conv2D(1024, (3, 3), activation='relu', padding='same')(pool4)  # Convolutional layer with 1024 filters
conv5 = BatchNormalization()(conv5)  # Apply batch normalization to normalize the activations
conv5 = Conv2D(1024, (3, 3), activation='relu', padding='same')(conv5)  # Convolutional layer with 1024 filters
conv5 = BatchNormalization()(conv5)  # Apply batch normalization to normalize the activations
drop5 = Dropout(0.5)(conv5)  # Apply dropout regularization with a rate of 0.5

# Decoder
up6 = concatenate([UpSampling2D((2, 2))(drop5), conv4], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv6 = Conv2D(512, (3, 3), activation='relu', padding='same')(up6)  # Convolutional layer with 512 filters
conv6 = BatchNormalization()(conv6)  # Apply batch normalization to normalize the activations
conv6 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv6)  # Convolutional layer with 512 filters
conv6 = BatchNormalization()(conv6)  # Apply batch normalization to normalize the activations

up7 = concatenate([UpSampling2D((2, 2))(conv6), conv3], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv7 = Conv2D(256, (3, 3), activation='relu', padding='same')(up7)  # Convolutional layer with 256 filters
conv7 = BatchNormalization()(conv7)  # Apply batch normalization to normalize the activations
conv7 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv7)  # Convolutional layer with 256 filters
conv7 = BatchNormalization()(conv7)  # Apply batch normalization to normalize the activations

up8 = concatenate([UpSampling2D((2, 2))(conv7), conv2], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv8 = Conv2D(128, (3, 3), activation='relu', padding='same')(up8)  # Convolutional layer with 128 filters
conv8 = BatchNormalization()(conv8)  # Apply batch normalization to normalize the activations
conv8 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv8)  # Convolutional layer with 128 filters
conv8 = BatchNormalization()(conv8)  # Apply batch normalization to normalize the activations

up9 = concatenate([UpSampling2D((2, 2))(conv8), conv1], axis=-1)  # Upsampling layer with a scale factor of (2, 2)
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same')(up9)  # Convolutional layer with 64 filters
conv9 = BatchNormalization()(conv9)  # Apply batch normalization to normalize the activations
conv9 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv9)  # Convolutional layer with 64 filters
conv9 = BatchNormalization()(conv9)  # Apply batch normalization to normalize the activations

outputs = Conv2D(n_classes, (1, 1), activation='softmax')(conv9)  # Convolutional layer for output

# Create the model
model = Model(inputs=inputs, outputs=outputs)

# Print the model summary
model.summary()


# Set the optimizer for the model
optimizer = tf.keras.optimizers.Adam(lr=1e-4)

# Compile the model with loss function and metrics
model.compile(optimizer=optimizer, loss='binary_crossentropy', metrics=['accuracy'])

images_path = '/kaggle/input/coco-2014-dataset-for-yolov3/coco2014/images/val2014'
masks_path = '/kaggle/working/mask_val_2014'
batch_size = 8

val_generator = CustomDataGenerator(images_path, masks_path, batch_size)



```


Also to ensure that all the input and output have proper shape I run the code below

```
def print_preprocessed_image_shapes(model, generator):
    """
    Print the shapes of preprocessed images generated by the provided model and generator.

    Args:
        model (tf.keras.Model): The trained model.
        generator (CustomDataGenerator): Instance of the CustomDataGenerator class.
    """
    for i in range(len(generator)):
        # Get a batch of preprocessed images from the generator
        batch_images, batch_masks = generator[i]

        # Print the shapes of the preprocessed images
        for image in batch_images:
            print(f"Shape of preprocessed image: {image.shape}")
        for mask in batch_maskss:

             print(f"Shape of preprocessed image: {mask.shape}")
            
# Print the shapes of preprocessed images
print_preprocessed_image_shapes(model, val_generator)
```
As a result of this error, the model was unable to undergo the training process.

### Standalone code to reproduce the issue

```shell
# Fit the model with the training generator
train_steps =  len(os.listdir( "/kaggle/working/mask_train_2014/"))/batch_size
model.fit(train_generator,validation_data = val_generator, steps_per_epoch = train_steps , epochs=20)
```


### Relevant log output

```shell
tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel_5/dropout_10/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```
