Dropout Layer training=True -- layout failed: INVALID_ARGUMENT
Hi @meekus-fischer ,

Apologies for the delay. This issue somehow skipped from my attention. I have tried replicating the issue but its executed without any problem.Please refer to attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/b45a66a7d19a7e01d5b61572fd5eb588/61687.ipynb#scrollTo=XBLXOPF3xdKV). Could you please cross check once and confirm.

Thank you! I just ran your gist on my system, and received the same error as before.  

```
Epoch 1/50
2023-10-03 07:23:36.765732: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inCustom_MC_Dropout_CNN/monte_carlo_dropout_22/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
935/938 [============================>.] - ETA: 0s - Accuracy: 0.9011 - Loss: 0.3918 - Precision: 0.3404 - Recall: 0.0217 - F1: 0.0408
2023-10-03 07:23:47.185562: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inCustom_MC_Dropout_CNN/monte_carlo_dropout_22/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```

One thing I noticed in your gist is that you were running with python 3.10 and I am running with 3.9; however, I am not sure the issue is related.

Again, I am running on NVIDIA RTX-8000 GPUs, the issue persists on A100s as well.  

Here is my Conda environment:

```
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
absl-py                   1.4.0                    pypi_0    pypi
anyio                     3.7.1              pyhd8ed1ab_0    conda-forge
argon2-cffi               23.1.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py39hb9d737c_3    conda-forge
arrow                     1.2.3              pyhd8ed1ab_0    conda-forge
asttokens                 2.2.1              pyhd8ed1ab_0    conda-forge
astunparse                1.6.3                    pypi_0    pypi
async-lru                 2.0.4              pyhd8ed1ab_0    conda-forge
attrs                     23.1.0             pyh71513ae_1    conda-forge
audioread                 3.0.0            py39hf3d152e_1    conda-forge
babel                     2.12.1             pyhd8ed1ab_1    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                pyhd8ed1ab_3    conda-forge
backports.functools_lru_cache 1.6.5              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.12.2             pyha770c72_0    conda-forge
bleach                    6.0.0              pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_9    conda-forge
brotli-bin                1.0.9                h166bdaf_9    conda-forge
brotli-python             1.0.9            py39h5a03fae_9    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2023.7.22            hbcca054_0    conda-forge
cached-property           1.5.2                hd8ed1ab_1    conda-forge
cached_property           1.5.2              pyha770c72_1    conda-forge
cachetools                5.3.1                    pypi_0    pypi
certifi                   2023.7.22                pypi_0    pypi
cffi                      1.15.1           py39he91dace_3    conda-forge
charset-normalizer        3.2.0              pyhd8ed1ab_0    conda-forge
cloudpickle               2.2.1                    pypi_0    pypi
comm                      0.1.4              pyhd8ed1ab_0    conda-forge
contourpy                 1.1.0            py39h7633fee_0    conda-forge
cuda-version              11.8                 h70ddcb2_2    conda-forge
cudatoolkit               11.8.0              h4ba93d1_12    conda-forge
cudnn                     8.8.0.121            h0800d71_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dbus                      1.13.18              hb2f20db_0  
debugpy                   1.6.8            py39h3d6467e_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dm-tree                   0.1.8                    pypi_0    pypi
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.3              pyhd8ed1ab_0    conda-forge
executing                 1.2.0              pyhd8ed1ab_0    conda-forge
expat                     2.5.0                hcb278e6_1    conda-forge
ffmpeg                    4.2.2                h20bf706_0  
flatbuffers               23.5.26                  pypi_0    pypi
fontconfig                2.14.2               h14ed4e7_0    conda-forge
fonttools                 4.42.1           py39hd1e30aa_0    conda-forge
fqdn                      1.5.1              pyhd8ed1ab_0    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
gast                      0.4.0                    pypi_0    pypi
gettext                   0.21.1               h27087fc_0    conda-forge
giflib                    5.2.1                h0b41bf4_3    conda-forge
glib                      2.76.4               hfc55251_0    conda-forge
glib-tools                2.76.4               hfc55251_0    conda-forge
gmp                       6.2.1                h58526e2_0    conda-forge
gnutls                    3.6.13               h85f3911_1    conda-forge
google-auth               2.22.0                   pypi_0    pypi
google-auth-oauthlib      1.0.0                    pypi_0    pypi
google-pasta              0.2.0                    pypi_0    pypi
grpcio                    1.57.0                   pypi_0    pypi
gst-plugins-base          1.14.1               h6a678d5_1  
gstreamer                 1.14.1               h5eee18b_1  
h5py                      3.9.0                    pypi_0    pypi
icu                       58.2              hf484d3e_1000    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.8.0              pyha770c72_0    conda-forge
importlib-resources       6.0.1              pyhd8ed1ab_0    conda-forge
importlib_metadata        6.8.0                hd8ed1ab_0    conda-forge
importlib_resources       6.0.1              pyhd8ed1ab_0    conda-forge
ipykernel                 6.25.1             pyh71e2992_0    conda-forge
ipython                   8.14.0             pyh41d4057_0    conda-forge
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                8.1.0              pyhd8ed1ab_0    conda-forge
isoduration               20.11.0            pyhd8ed1ab_0    conda-forge
jedi                      0.19.0             pyhd8ed1ab_0    conda-forge
jinja2                    3.1.2              pyhd8ed1ab_1    conda-forge
joblib                    1.3.2              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h0b41bf4_3    conda-forge
json5                     0.9.14             pyhd8ed1ab_0    conda-forge
jsonpointer               2.0                        py_0    conda-forge
jsonschema                4.19.0             pyhd8ed1ab_1    conda-forge
jsonschema-specifications 2023.7.1           pyhd8ed1ab_0    conda-forge
jsonschema-with-format-nongpl 4.19.0             pyhd8ed1ab_1    conda-forge
jupyter                   1.0.0            py39hf3d152e_8    conda-forge
jupyter-lsp               2.2.0              pyhd8ed1ab_0    conda-forge
jupyter_client            8.3.0              pyhd8ed1ab_0    conda-forge
jupyter_console           6.6.3              pyhd8ed1ab_0    conda-forge
jupyter_core              5.3.1            py39hf3d152e_0    conda-forge
jupyter_events            0.7.0              pyhd8ed1ab_2    conda-forge
jupyter_server            2.7.1              pyhd8ed1ab_0    conda-forge
jupyter_server_terminals  0.4.4              pyhd8ed1ab_1    conda-forge
jupyterlab                4.0.5              pyhd8ed1ab_0    conda-forge
jupyterlab_pygments       0.2.2              pyhd8ed1ab_0    conda-forge
jupyterlab_server         2.24.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        3.0.8              pyhd8ed1ab_0    conda-forge
keras                     2.13.1                   pypi_0    pypi
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.4            py39hf939315_1    conda-forge
krb5                      1.20.1               h81ceb04_0    conda-forge
lame                      3.100             h166bdaf_1003    conda-forge
lazy_loader               0.3                pyhd8ed1ab_0    conda-forge
lcms2                     2.15                 hfd0df8a_0    conda-forge
ld_impl_linux-64          2.40                 h41732ed_0    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           17_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_9    conda-forge
libbrotlidec              1.0.9                h166bdaf_9    conda-forge
libbrotlienc              1.0.9                h166bdaf_9    conda-forge
libcblas                  3.9.0           17_linux64_openblas    conda-forge
libclang                  16.0.6                   pypi_0    pypi
libdeflate                1.17                 h5eee18b_0  
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libexpat                  2.5.0                hcb278e6_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libflac                   1.4.3                h59595ed_0    conda-forge
libgcc-ng                 13.1.0               he5830b7_0    conda-forge
libgfortran-ng            13.1.0               h69a702a_0    conda-forge
libgfortran5              13.1.0               h15d22d2_0    conda-forge
libglib                   2.76.4               hebfc3b9_0    conda-forge
libgomp                   13.1.0               he5830b7_0    conda-forge
libiconv                  1.17                 h166bdaf_0    conda-forge
liblapack                 3.9.0           17_linux64_openblas    conda-forge
libllvm10                 10.0.1               he513fc3_3    conda-forge
libllvm14                 14.0.6               hcd5def8_4    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libogg                    1.3.4                h7f98852_1    conda-forge
libopenblas               0.3.23          pthreads_h80387f5_0    conda-forge
libopus                   1.3.1                h7f98852_1    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libpq                     12.15                hdbd6064_1  
librosa                   0.10.1             pyhd8ed1ab_0    conda-forge
libsndfile                1.2.2                hbc2eb40_0    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsqlite                 3.42.0               h2797004_0    conda-forge
libstdcxx-ng              13.1.0               hfd8a6a1_0    conda-forge
libtiff                   4.5.0                h6adf6a1_2    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libvorbis                 1.3.7                h9c3ff4c_0    conda-forge
libvpx                    1.7.0                h439df22_0  
libwebp                   1.2.4                h1daa5a0_1    conda-forge
libwebp-base              1.2.4                h5eee18b_1  
libxcb                    1.15                 h0b41bf4_0    conda-forge
libxkbcommon              1.0.1                hfa300c1_0  
libxml2                   2.9.14               h74e7548_0  
libxslt                   1.1.35               h4e12654_0  
libzlib                   1.2.13               hd590300_5    conda-forge
llvmlite                  0.40.1           py39h174d805_0    conda-forge
markdown                  3.4.4                    pypi_0    pypi
markupsafe                2.1.3            py39hd1e30aa_0    conda-forge
matplotlib                3.7.2            py39hf3d152e_0    conda-forge
matplotlib-base           3.7.2            py39h0126182_0    conda-forge
matplotlib-inline         0.1.6              pyhd8ed1ab_0    conda-forge
mistune                   3.0.1              pyhd8ed1ab_0    conda-forge
mpg123                    1.31.3               hcb278e6_0    conda-forge
msgpack-python            1.0.5            py39h4b4f3f3_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbclient                  0.8.0              pyhd8ed1ab_0    conda-forge
nbconvert                 7.7.4              pyhd8ed1ab_0    conda-forge
nbconvert-core            7.7.4              pyhd8ed1ab_0    conda-forge
nbconvert-pandoc          7.7.4              pyhd8ed1ab_0    conda-forge
nbformat                  5.9.2              pyhd8ed1ab_0    conda-forge
ncurses                   6.4                  hcb278e6_0    conda-forge
nest-asyncio              1.5.6              pyhd8ed1ab_0    conda-forge
nettle                    3.6                  he412f7d_0    conda-forge
notebook                  7.0.2              pyhd8ed1ab_0    conda-forge
notebook-shim             0.2.3              pyhd8ed1ab_0    conda-forge
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.92                 h1d7d5a4_0    conda-forge
numba                     0.57.1           py39hb75a051_0    conda-forge
numpy                     1.24.3                   pypi_0    pypi
oauthlib                  3.2.2                    pypi_0    pypi
openh264                  2.1.1                h4ff587b_0  
openssl                   3.1.2                hd590300_0    conda-forge
opt-einsum                3.3.0                    pypi_0    pypi
overrides                 7.4.0              pyhd8ed1ab_0    conda-forge
packaging                 23.1               pyhd8ed1ab_0    conda-forge
pandas                    2.0.3            py39h40cae4c_1    conda-forge
pandoc                    3.1.3                h32600fe_0    conda-forge
pandocfilters             1.5.0              pyhd8ed1ab_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
patsy                     0.5.3              pyhd8ed1ab_0    conda-forge
pcre2                     10.40                hc3806b6_0    conda-forge
pexpect                   4.8.0              pyh1a96a4e_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.4.0            py39h6a678d5_0  
pip                       23.2.1             pyhd8ed1ab_0    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_0    conda-forge
platformdirs              3.10.0             pyhd8ed1ab_0    conda-forge
ply                       3.11                       py_1    conda-forge
pooch                     1.7.0              pyha770c72_3    conda-forge
prometheus_client         0.17.1             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.39             pyha770c72_0    conda-forge
prompt_toolkit            3.0.39               hd8ed1ab_0    conda-forge
protobuf                  4.24.1                   pypi_0    pypi
psutil                    5.9.5            py39h72bdee0_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
pure_eval                 0.2.2              pyhd8ed1ab_0    conda-forge
pyasn1                    0.5.0                    pypi_0    pypi
pyasn1-modules            0.3.0                    pypi_0    pypi
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.16.1             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pyqt                      5.15.7           py39h6a678d5_1  
pyqt5-sip                 12.11.0          py39h6a678d5_1  
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pysoundfile               0.12.1             pyhd8ed1ab_0    conda-forge
python                    3.9.17          h0755675_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-fastjsonschema     2.18.0             pyhd8ed1ab_0    conda-forge
python-json-logger        2.0.7              pyhd8ed1ab_0    conda-forge
python-tzdata             2023.3             pyhd8ed1ab_0    conda-forge
python_abi                3.9                      3_cp39    conda-forge
pytz                      2023.3             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0.1            py39hd1e30aa_0    conda-forge
pyzmq                     25.1.1           py39hb257651_0    conda-forge
qt-main                   5.15.2               h327a75a_7  
qt-webengine              5.15.9               hd2b0992_4  
qtconsole                 5.4.3              pyhd8ed1ab_0    conda-forge
qtconsole-base            5.4.3              pyha770c72_0    conda-forge
qtpy                      2.3.1              pyhd8ed1ab_0    conda-forge
qtwebkit                  5.212                h4eab89a_4  
readline                  8.2                  h8228510_1    conda-forge
referencing               0.30.2             pyhd8ed1ab_0    conda-forge
requests                  2.31.0             pyhd8ed1ab_0    conda-forge
requests-oauthlib         1.3.1                    pypi_0    pypi
rfc3339-validator         0.1.4              pyhd8ed1ab_0    conda-forge
rfc3986-validator         0.1.1              pyh9f0ad1d_0    conda-forge
rpds-py                   0.9.2            py39h9fdd4d6_0    conda-forge
rsa                       4.9                      pypi_0    pypi
scikit-learn              1.3.0            py39hc236052_0    conda-forge
scipy                     1.11.2           py39h6183b62_0    conda-forge
seaborn                   0.12.2               hd8ed1ab_0    conda-forge
seaborn-base              0.12.2             pyhd8ed1ab_0    conda-forge
send2trash                1.8.2              pyh41d4057_0    conda-forge
setuptools                68.1.2             pyhd8ed1ab_0    conda-forge
sip                       6.6.2            py39h6a678d5_0  
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.3.0              pyhd8ed1ab_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
soxr                      0.1.3                h0b41bf4_3    conda-forge
soxr-python               0.3.5            py39h0f8d45d_0    conda-forge
sqlite                    3.42.0               h2c6b66d_0    conda-forge
stack_data                0.6.2              pyhd8ed1ab_0    conda-forge
statsmodels               0.14.0           py39h0f8d45d_1    conda-forge
tensorboard               2.13.0                   pypi_0    pypi
tensorboard-data-server   0.7.1                    pypi_0    pypi
tensorflow                2.13.0                   pypi_0    pypi
tensorflow-estimator      2.13.0                   pypi_0    pypi
tensorflow-io-gcs-filesystem 0.33.0                   pypi_0    pypi
tensorflow-probability    0.21.0                   pypi_0    pypi
termcolor                 2.3.0                    pypi_0    pypi
terminado                 0.17.1             pyh41d4057_0    conda-forge
threadpoolctl             3.2.0              pyha21a80b_0    conda-forge
tinycss2                  1.2.1              pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
toml                      0.10.2             pyhd8ed1ab_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tornado                   6.3.3            py39hd1e30aa_0    conda-forge
traitlets                 5.9.0              pyhd8ed1ab_0    conda-forge
typing-extensions         4.5.0                    pypi_0    pypi
typing_extensions         4.7.1              pyha770c72_0    conda-forge
typing_utils              0.1.0              pyhd8ed1ab_0    conda-forge
tzdata                    2023c                h71feb2d_0    conda-forge
unicodedata2              15.0.0           py39hb9d737c_0    conda-forge
uri-template              1.3.0              pyhd8ed1ab_0    conda-forge
urllib3                   1.26.16                  pypi_0    pypi
wcwidth                   0.2.6              pyhd8ed1ab_0    conda-forge
webcolors                 1.13               pyhd8ed1ab_0    conda-forge
webencodings              0.5.1                      py_1    conda-forge
websocket-client          1.6.2              pyhd8ed1ab_0    conda-forge
werkzeug                  2.3.7                    pypi_0    pypi
wheel                     0.41.2             pyhd8ed1ab_0    conda-forge
widgetsnbextension        4.0.8              pyhd8ed1ab_0    conda-forge
wrapt                     1.15.0                   pypi_0    pypi
x264                      1!157.20191217       h7b6447c_0  
xorg-libxau               1.0.11               hd590300_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.16.2             pyhd8ed1ab_0    conda-forge
zlib                      1.2.13               hd590300_5    conda-forge
zstd                      1.5.2                hfc55251_7    conda-forge
``` @SuryanarayanaY any update on this? I'm getting some similar warnings/errors. TF 2.14 with a small custom model, on a machine with a single GPU.

```
2023-11-02 16:48:26.266488: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:961] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape insequential/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer

...

2023-11-02 16:48:27.584269: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 12961131670770205513
2023-11-02 16:48:27.584277: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 5985745776991298471
2023-11-02 16:48:27.676188: I tensorflow/core/framework/local_rendezvous.cc:421] Local rendezvous recv item cancelled. Key hash: 17452564472235251298
```

Only a couple of the "Local rendezvous recv item cancelled" messages before the model starts training, at which point it seems to do okay. I got rid of the first error/warning by turning off the layout optimizer, as mentioned at the bottom of [this thread](https://github.com/tensorflow/tensorflow/issues/34499). I also got the same error, Can anyone help me?

```python
2023-11-08 17:48:04.891105: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 8295284736 exceeds 10% of free system memory.
2023-11-08 17:48:08.059367: W tensorflow/tsl/framework/cpu_allocator_impl.cc:83] Allocation of 8295284736 exceeds 10% of free system memory.
Epoch 1/100
2023-11-08 17:48:11.436880: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:961] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel/dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
2023-11-08 17:48:11.812426: I tensorflow/tsl/platform/default/subprocess.cc:304] Start cannot spawn child process: No such file or directory
```

**`The Kernel crashed while executing code in the the current cell or a previous cell. Please review the code in the cell(s) to identify a possible cause of the failure. Click [here](https://aka.ms/vscodeJupyterKernelCrash) for more info. View Jupyter [log](command:jupyter.viewOutput) for further details.`** Error also persist in tensorflow `2.14.0` version! Hi @meekus-fischer ,

I have tested with TF2.14V on colab GPU(single GPU) runtime and it works fine. Attached [gist](https://colab.sandbox.google.com/gist/SuryanarayanaY/fcf32c75cbf79e132ef9a5be004766de/61687-tf2-14v.ipynb) for reference.

Please make sure to install `tensorflow[and-cuda]` package for GPU support. Please cross check the gist and confirm.

Please also check whether the issue is on multi GPU environment.  Hi @SuryanarayanaY, I have tested on Windows WSL2 on version 2.14.0 and the error persists. I also have the single GPU support. @iamtekson ,

Could you please provide a gist to have a look into it.Please confirm whether you are using single GPU setup or Multi GPU ?

Thanks! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61687">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/61687">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

binary

### TensorFlow version

2.13.0

### Custom code

Yes

### OS platform and distribution

RHELS 7.9

### Mobile device

_No response_

### Python version

3.9

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

11.8 / 8.8

### GPU model and memory

NVIDIA A100-SXM4-40GB, Compute Capability 8.0

### Current behavior?

```
2023-08-24 11:01:15.073207: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inCustom_MC_Dropout_CNN/monte_carlo_dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
```

This arises from keras Dropout layer with training set to True for MC Dropout implementation.  Unclear why this happens in 2.13 and does not happen in previous versions of Tensorflow.  

gist can be found [here](https://gist.github.com/meekus-fischer/46d92232d9c503be3bfa76f225315b72)

### Standalone code to reproduce the issue

```shell
Custom model doing binary classification.  

gist can be found [here](https://gist.github.com/meekus-fischer/46d92232d9c503be3bfa76f225315b72)
```


### Relevant log output

```shell
Epoch 1/2000
2023-08-24 11:01:15.073207: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inCustom_MC_Dropout_CNN/monte_carlo_dropout/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer
2023-08-24 11:01:18.489988: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8800
2023-08-24 11:01:19.296731: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8800
2023-08-24 11:01:20.504944: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8800
2023-08-24 11:01:21.410170: I tensorflow/compiler/xla/stream_executor/cuda/cuda_dnn.cc:432] Loaded cuDNN version 8800
2023-08-24 11:01:22.693953: I tensorflow/compiler/xla/stream_executor/cuda/cuda_blas.cc:606] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
NCCL version 2.13.4+cudaCUDA_MAJOR.CUDA_MINOR
2023-08-24 11:01:25.775749: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f0008dbb5b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2023-08-24 11:01:25.775881: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): NVIDIA A100-SXM4-40GB, Compute Capability 8.0
2023-08-24 11:01:25.775907: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (1): NVIDIA A100-SXM4-40GB, Compute Capability 8.0
2023-08-24 11:01:25.775920: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (2): NVIDIA A100-SXM4-40GB, Compute Capability 8.0
2023-08-24 11:01:25.775931: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (3): NVIDIA A100-SXM4-40GB, Compute Capability 8.0
2023-08-24 11:01:25.786715: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:255] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-08-24 11:01:25.786798: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:255] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2023-08-24 11:01:26.258524: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.
  10622/Unknown - 346s 30ms/step - Accuracy: 0.8316 - Loss: 0.2066 - Precision: 0.0749 - Recall: 0.5010 - F1: 0.13022023-08-24 11:06:50.967839: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10230612448716145652
2023-08-24 11:06:50.968030: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15407084872456274683
2023-08-24 11:06:50.968252: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17592376860366209384
2023-08-24 11:06:50.968315: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 59586944474759984
2023-08-24 11:06:50.968338: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4843790225391952165
2023-08-24 11:06:50.968526: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5723252417712899607
2023-08-24 11:06:50.968555: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7116492272812476647
2023-08-24 11:06:50.968575: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6169583800350491318
2023-08-24 11:06:50.968603: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14279008608205112804
2023-08-24 11:06:50.968624: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2035256671229577509
2023-08-24 11:06:50.968637: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16934632576416694905
2023-08-24 11:06:50.968693: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3559188353333742787
2023-08-24 11:06:50.968719: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15823058833201008187
2023-08-24 11:06:50.968737: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14146042709357596467
2023-08-24 11:06:50.968754: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7736629887233675932
2023-08-24 11:06:50.968772: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8429871751161250943
2023-08-24 11:06:50.968797: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10517342425574574684
2023-08-24 11:06:50.968818: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6970197396290489839
2023-08-24 11:06:50.968831: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4931681574040413300
2023-08-24 11:06:50.968875: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 626925344230971222
2023-08-24 11:06:50.968901: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7470810518621668803
2023-08-24 11:06:50.968925: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12385602897828204054
2023-08-24 11:06:50.968946: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 18188314560958681369
2023-08-24 11:06:50.968958: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13624849890299536610
2023-08-24 11:06:50.969001: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17807137559239481777
2023-08-24 11:06:50.969025: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2139352706329042097
2023-08-24 11:06:50.969044: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1640340635315586973
2023-08-24 11:06:50.969069: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6223702484981372229
2023-08-24 11:06:50.969089: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10172410999163213764
2023-08-24 11:06:50.969122: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6497789849068078695
2023-08-24 11:06:50.969166: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14547651345659747702
2023-08-24 11:06:50.969193: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17623891083240824387
2023-08-24 11:06:50.969211: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13064636275713549317
2023-08-24 11:06:50.969228: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12396583141502166397
2023-08-24 11:06:50.969246: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5913395196567159933
2023-08-24 11:06:50.969271: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13627801007941572809
2023-08-24 11:06:50.969293: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14421550386438117999
2023-08-24 11:06:50.969307: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17275079556186662745
2023-08-24 11:06:50.969351: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17233527917734094487
2023-08-24 11:06:50.969377: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1559338696184663628
2023-08-24 11:06:50.969400: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5466597647379635179
2023-08-24 11:06:50.969419: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3733731331292716199
2023-08-24 11:06:50.969433: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1612774677619861335
2023-08-24 11:06:50.969477: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8582450577167281147
2023-08-24 11:06:50.969501: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2782820814590454977
2023-08-24 11:06:50.969519: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3393985160674389282
2023-08-24 11:06:50.969537: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10933067372207001568
2023-08-24 11:06:50.969561: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12568695295317122124
2023-08-24 11:06:50.969581: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9541194832674758364
2023-08-24 11:06:50.969594: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8581202827114056180
2023-08-24 11:06:50.969638: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6233296683631670242
2023-08-24 11:06:50.969672: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9668322976265270062
2023-08-24 11:06:50.969689: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14184838215839790752
2023-08-24 11:06:50.969706: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7224663333712770602
2023-08-24 11:06:50.969723: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7397044618240997942
2023-08-24 11:06:50.969740: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8130222007033543507
2023-08-24 11:06:50.969784: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9705492144130363178
2023-08-24 11:06:50.969807: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16679483023940330274
2023-08-24 11:06:50.969828: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3293157344159655099
2023-08-24 11:06:50.969841: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16377046709051984690
2023-08-24 11:06:50.969887: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15402017112026381377
2023-08-24 11:06:50.969912: I tensorflow/core/framework/local_rendezvous.cc:409] Local rendezvous send item cancelled. Key hash: 9526661262069731973
2023-08-24 11:06:50.969934: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13997955057405387944
2023-08-24 11:06:50.969954: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7535581894821128379
2023-08-24 11:06:50.969967: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8329440098288561250
2023-08-24 11:06:50.970010: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9064305880271633177
2023-08-24 11:06:50.970033: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10920339178713879663
2023-08-24 11:06:50.970054: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17331780224083848384
2023-08-24 11:06:50.970073: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4387767839775239829
2023-08-24 11:06:50.970087: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14235745259444133525
2023-08-24 11:06:50.970129: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15121875451797684147
2023-08-24 11:06:50.970156: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2086183684301986528
2023-08-24 11:06:50.970177: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13396972508038279194
2023-08-24 11:06:50.970197: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3769933078929813764
2023-08-24 11:06:50.970210: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9191688026028395470
2023-08-24 11:06:50.970257: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16698476566770966153
2023-08-24 11:06:50.970278: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14753529190433239649
2023-08-24 11:06:50.970298: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13900114517410444549
2023-08-24 11:06:50.970314: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2047052490661242150
2023-08-24 11:06:50.970342: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11154800211652874765
2023-08-24 11:06:50.970361: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9142657018531009635
2023-08-24 11:06:50.970377: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5427900937422839708
2023-08-24 11:06:50.970398: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16660179110683671619
2023-08-24 11:06:50.970415: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12852018438355285325
2023-08-24 11:06:50.970464: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 607352058025025138
2023-08-24 11:06:50.970486: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5852029023401935321
2023-08-24 11:06:50.970504: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3708094119504533135
2023-08-24 11:06:50.970528: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12012444379425185333
2023-08-24 11:06:50.970548: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2694129016930453301
2023-08-24 11:06:50.970565: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16690881604261796875
2023-08-24 11:06:50.970588: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3200238774805194023
2023-08-24 11:06:50.970606: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12303021566759651641
2023-08-24 11:06:50.970623: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12559606935649894050
2023-08-24 11:06:50.970652: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14073952303962254143
2023-08-24 11:06:50.970672: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12728231451437746801
2023-08-24 11:06:50.970689: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 275694950219046675
2023-08-24 11:06:50.970712: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8794029419552633053
2023-08-24 11:06:50.970731: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10700741266189635836
2023-08-24 11:06:50.970747: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12210410409207542263
2023-08-24 11:06:50.970761: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1426965020033241286
2023-08-24 11:06:50.970776: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14005291578614903458
2023-08-24 11:06:50.970789: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7129919436717380486
2023-08-24 11:06:50.970804: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 305166255698457613
2023-08-24 11:06:50.970818: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2298180445646427191
2023-08-24 11:06:50.970832: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2169690745073579443
2023-08-24 11:06:50.970846: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4743653862056645976
2023-08-24 11:06:50.970860: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13014453433668203963
2023-08-24 11:06:50.970874: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9907502355052484286
2023-08-24 11:06:50.970888: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14023629169546197118
2023-08-24 11:06:50.970902: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8147796953383623256
2023-08-24 11:06:50.970916: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11731520165517458923
2023-08-24 11:06:50.970940: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13054994235575065371
2023-08-24 11:06:50.970959: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9747176552551324986
2023-08-24 11:06:50.970973: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17117647137538334341
2023-08-24 11:06:50.970989: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5153611993980018316
2023-08-24 11:06:50.971003: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15004025391910118056
2023-08-24 11:06:50.971018: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3063790399388444401
2023-08-24 11:06:50.971052: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3830525127148040556
2023-08-24 11:06:50.971068: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4749045392559338476
2023-08-24 11:06:50.971115: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16071133848988202381
2023-08-24 11:06:50.971131: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17981742675600701201
2023-08-24 11:06:50.971146: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17623561159131070056
2023-08-24 11:06:50.971168: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15285201522450726881
2023-08-24 11:07:29.289624: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4075359580615049412
2023-08-24 11:07:29.289849: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15086777354078093967
2023-08-24 11:07:29.289868: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7764170359290675737
2023-08-24 11:07:29.289886: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4281287878382289967
2023-08-24 11:07:29.289902: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15716950925811516672
2023-08-24 11:07:29.289920: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 18437880324272310277
2023-08-24 11:07:29.289936: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 84132605581087162
2023-08-24 11:07:29.289954: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9426939307156020373
2023-08-24 11:07:29.290198: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2146862422121390738
2023-08-24 11:07:29.290235: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3462541954341437694
2023-08-24 11:07:29.290430: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4757251772246970932
2023-08-24 11:07:29.290456: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2310900862945371053
2023-08-24 11:07:29.290496: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8093777233699734653
2023-08-24 11:07:29.290526: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3771959543230644229
2023-08-24 11:07:29.290571: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5900557290590721187
2023-08-24 11:07:29.290597: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2455567737869551008
2023-08-24 11:07:29.290686: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6098076008741815878
2023-08-24 11:07:29.290723: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1553848879136391844
2023-08-24 11:07:29.290769: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13917212505871489327
2023-08-24 11:07:29.290795: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9809958572552914958
2023-08-24 11:07:29.290821: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 198252566913829033
2023-08-24 11:07:29.290837: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5073347325328566998
2023-08-24 11:07:29.290855: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5978006840724482833
2023-08-24 11:07:29.290872: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12595934042829111589
2023-08-24 11:07:29.290916: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9387251194334273292
2023-08-24 11:07:29.290940: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16968343877100670238
2023-08-24 11:07:29.290963: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15297105432650139094
2023-08-24 11:07:29.290990: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4081987408384352687
2023-08-24 11:07:29.291016: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5493947754659818087
2023-08-24 11:07:29.291028: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6850845529302741243
2023-08-24 11:07:29.291071: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15022677012545259154
2023-08-24 11:07:29.291096: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13233726862595811952
2023-08-24 11:07:29.291123: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14234134314023218357
2023-08-24 11:07:29.291139: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12483608611790893202
2023-08-24 11:07:29.291156: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12148270049744368847
2023-08-24 11:07:29.291171: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5321921797854171573
2023-08-24 11:07:29.291197: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1832801212251014439
2023-08-24 11:07:29.291222: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8390992797373478667
2023-08-24 11:07:29.291245: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15510758115605779543
2023-08-24 11:07:29.291271: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5470046073078944888
2023-08-24 11:07:29.291283: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14658565871101820964
2023-08-24 11:07:29.291327: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14317614836688813112
2023-08-24 11:07:29.291351: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6236598275352729223
2023-08-24 11:07:29.291384: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8714587429891558384
2023-08-24 11:07:29.291412: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11498223346710789742
2023-08-24 11:07:29.291435: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 14476188790886421798
2023-08-24 11:07:29.291458: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12475594966318812074
2023-08-24 11:07:29.291483: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11601898139673393738
2023-08-24 11:07:29.291506: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4976030311990886841
2023-08-24 11:07:29.291529: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6617209893611589109
2023-08-24 11:07:29.291551: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13177787027967964473
2023-08-24 11:07:29.291574: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16943892717213549101
2023-08-24 11:07:29.291596: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2223833743739703384
2023-08-24 11:07:29.291618: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10219062227532962953
2023-08-24 11:07:29.291645: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13724040127314953106
2023-08-24 11:07:29.291668: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2067624533502072576
2023-08-24 11:07:29.291684: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2693354250884363784
2023-08-24 11:07:29.291707: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5384013371876801148
2023-08-24 11:07:29.291731: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16838618676439021446
2023-08-24 11:07:29.291753: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8576730697822492479
2023-08-24 11:07:29.291776: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7205413612177464488
2023-08-24 11:07:29.291798: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7419410273381722556
2023-08-24 11:07:29.291820: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12387571022272485464
2023-08-24 11:07:29.291842: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10096942157971947615
2023-08-24 11:07:29.291865: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17235744156686445141
2023-08-24 11:07:29.291886: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7157660774755047909
2023-08-24 11:07:29.291908: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 3223238886696215390
2023-08-24 11:07:29.291930: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6959224094144924436
2023-08-24 11:07:29.291952: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 320303901872528994
2023-08-24 11:07:29.291974: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4678411222501451155
2023-08-24 11:07:29.292004: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11466697275319965486
2023-08-24 11:07:29.292027: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16940283489605771424
2023-08-24 11:07:29.292050: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4318812230278670660
2023-08-24 11:07:29.292074: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10793570788563462816
2023-08-24 11:07:29.292096: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16389789610319880674
2023-08-24 11:07:29.292118: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15412920732005214795
2023-08-24 11:07:29.292141: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9706362058363342851
2023-08-24 11:07:29.292162: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 282440926482311630
2023-08-24 11:07:29.292184: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16604411440337789159
2023-08-24 11:07:29.292209: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2305311315300667846
2023-08-24 11:07:29.292231: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 13471908878414400865
2023-08-24 11:07:29.292252: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5920473717615552089
2023-08-24 11:07:29.292276: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 17321862789316030721
2023-08-24 11:07:29.292297: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 230923248894930545
2023-08-24 11:07:29.292319: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9790266787664553869
2023-08-24 11:07:29.292341: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8329428023832785896
2023-08-24 11:07:29.292364: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7908458004814008345
2023-08-24 11:07:29.292385: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4993197094509241984
2023-08-24 11:07:29.292407: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4091926352704099496
2023-08-24 11:07:29.292429: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8630806193082773190
2023-08-24 11:07:29.292451: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 1510013706380373244
2023-08-24 11:07:29.292473: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7061673450255603616
2023-08-24 11:07:29.292496: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 16228952560203272745
2023-08-24 11:07:29.292517: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15134061215888175049
2023-08-24 11:07:29.292539: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 2181129643598772086
2023-08-24 11:07:29.292561: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 12275612442691882545
2023-08-24 11:07:29.292582: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 11044755321133213194
2023-08-24 11:07:29.292604: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 6335108143304021966
2023-08-24 11:07:29.292644: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 5901561706985968925
2023-08-24 11:07:29.292668: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15343645954539090124
2023-08-24 11:07:29.292692: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7889586265790665191
2023-08-24 11:07:29.292719: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9687346430831728350
2023-08-24 11:07:29.292741: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 10459441727503677044
2023-08-24 11:07:29.292764: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9104442354481575726
2023-08-24 11:07:29.292786: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15547032933648534830
2023-08-24 11:07:29.292807: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7890544443613869694
2023-08-24 11:07:29.292830: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 4123678406397018076
2023-08-24 11:07:29.292852: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 7440280401194482935
2023-08-24 11:07:29.292875: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 8646180232994160174
2023-08-24 11:07:29.292894: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 15193230168019467790
2023-08-24 11:07:29.292917: I tensorflow/core/framework/local_rendezvous.cc:405] Local rendezvous recv item cancelled. Key hash: 9199876006741229390
```
```
