Unable to run model.fit() in WSL environment
![Issue](https://user-images.githubusercontent.com/132931947/236890298-8cb15098-86bd-403e-b20a-7cec6a7ed140.png)
 @TP066335,
I was facing a different error while executing the given code. Kindly find the gist of it [here](https://colab.research.google.com/gist/tilakrayal/0066ddfd8c3aed29b5301667bb9dce44/untitled1122.ipynb) and provide complete dependencies to analyse the issue and complete error as well.  

Also if you are trying to execute the code on multi-gpu, Usually when you run the multi GPU distributed training in the eager mode, for each GPU it creates a new session and these sessions will not be synchronized since these are running in the eager mode and executes immediately.
Even though `tf.distribute.Srategy` works well with both **eager** and **tf.function**, it works best with tf.function and eager mode is recommended for debug mode.                                                     I was getting same error on my single GPU also while trying model.fit(). However it worked perfectly on my cpu and google collab GPU. Not really sure why I was getting that error. Same GPU and memory as his. Followed the installation from this: https://www.tensorflow.org/install/pip#windows-wsl2 Hi @tilakrayal ,

I check your gist, actually you can remove that code because Google Colab needs to install scikeras separately. My issue is not with Google Colab, it is related to WSL in my own machine. 

Besides, I'm using single gpu, not multi-gpu.

Steps:
1. I setup my GPU for tensorflow according to this video, you may refer to this video for all my dependecies. https://www.youtube.com/watch?v=KinTNHO-6IY
3. You may get the complete error from my attached IPNYB file. 
4. I also attached a small dataset for your debugging. 

[data.zip](https://github.com/tensorflow/tensorflow/files/11431804/data.zip)

Hi @eimiss, yes, I'm having the same problem as yours.  Hi @tilakrayal,

Fyi, you may use my IPNYB file (data.zip) for debugging because I had simplified the codes to help your debugging.

Besides, can you help me to remove "DL_TP066335_xxxx.docx". I accidentally uploaded the document here, I only able to remove the link, but not the physical document. @TP066335,
I request you to take a look at this [issue](https://github.com/tensorflow/tensorflow/issues/59779) and comment from the [developer](https://github.com/tensorflow/tensorflow/issues/59779#issuecomment-1540864610) where users are facing similar error and the issue is still open.Also I request to follow the similar issue which has been proposed to have the updates on the similar issue.Thank you! @tilakrayal,

If you read their error message, you will notice their case is different from mine case. Besides, they are not using GPU. 

Please ignore the rest of the alerts in my IPNYB file, because those aleart messages are common when using GPU.

The only problem is the last error message which stop my training.
2023-05-09 01:04:47.558215: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int32
	 [[{{node Placeholder/_0}}]]

Please take note that I have no issue running the same code using my CPU in windows environment, the error message prompted when I'm using GPU to train the model in WSL environment. 

Below is the statement from Tensorflow. In other words, we have no choice but to use WSL in order to use GPU. 
"Starting with TensorFlow 2.11, you will need to install [TensorFlow in WSL2](https://tensorflow.org/install/pip#windows-wsl2)."

If you follow the same stesps to setup WSL, most likely you will get the same error when running my Python code in WSL environment.





 Yeah I also fixed it, thanks. @eimiss,

It would be greatly appreciated if you could share with me the steps on how to fix the issue. I uninstalled everything and followed this guide:
https://www.youtube.com/watch?v=ttxtV966jyQ
then I isntalled tensorflow as it's shown in the tensorflow page Thanks @eimiss.

The method to uninstalled everything not feasible for my machine.
Maybe I will wait for @tilakrayal for solution on WSL. Hi @TP066335 ,

Could you please share the outputs of following commands. I want to ensure whether GPU driver and CUDA path has been set or not properly.

`nvidia-smi`

`python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"` Hi @SuryanarayanaY,

Thank you so much for your kind attention to this issue. 
Please find the attached screenshot as per requested.

[Doc1.docx](https://github.com/tensorflow/tensorflow/files/11508259/Doc1.docx)
 Hi. I am having the same issue as @TP066335. I get the same result running model.fit() as shown in the screenshot. I also followed the same instructions from tensorflow.org about how to install for Windows WSL2.

I am submitting screenshots since my GPU and drivers are not quite the same. It appears to be a problem with running a GPU under WSL2.
![Screenshot 2023-05-25 211001](https://github.com/tensorflow/tensorflow/assets/289085/ad3d4da6-87c8-4a2d-a4a1-683e7c94f959)

![Screenshot 2023-05-25 211607](https://github.com/tensorflow/tensorflow/assets/289085/a3276e18-f4ce-4153-9666-a670e4665651)

Hope this helps. Thank you for your attention to this. I look forward to running TensorFlow on this GPU.
 Hi @TP066335 , 

Could you please try to import keras  as a backend to Tensorflow .

```
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
```

Let us know the outcome. Thanks!
 Hi @sachinprasadhs ,

Thank you so much for your kind assistance, I really appreciate it.

It appears that the issue is still stuck at the model.fit() step.

I have attached the ipynb file for yor reference.

Thank you.

[GPU.zip](https://github.com/tensorflow/tensorflow/files/12102117/GPU.zip) Hi @TP066335 Are you still stuck. Search "WSL libcuda is not a symbolic link" Hi @forthmedia, 

Thanks for the suggestion.

I follow the instructions from this [link ](https://superuser.com/questions/1707681/wsl-libcuda-is-not-a-symbolic-link) with regards to "WSL libcuda is not a symbolic link". I replaced the libcuda.so and libcuda.so.1 from C:\Windows\System32\lxss\lib.

I'm pleased to share that I've made significant progress, as I am now **able to execute the model.fit() function**. However, an unfortunate **challenge remains - the GPU is not being utilized during the process**, as depicted in the attached screenshot.

In conclusion, I am still unable to leverage the GPU for efficient model training.

Thank you.

[GPU Screenshot.zip](https://github.com/tensorflow/tensorflow/files/12329044/GPU.Screenshot.zip)



 Hello All,

I have successfully resolved the issue by following the comprehensive steps provided by KGP Talkie in their tutorial on [YouTube](https://www.youtube.com/watch?v=OHCPGvf06EA), which was posted two weeks ago. 

Additionally, to enhance the solution, I executed the command "conda install -c conda-forge cudatoolkit=11.2 cudnn=8.1.0". As a result, my CNN training can run perfectly within the WSL environment.

Last but not least, thanks a lot for your kind assistance and support. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60537">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/60537">No</a>

<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Bug

### Have you reproduced the bug with TF nightly?

Yes

### Source

binary

### Tensorflow Version

2.12.0

### Custom Code

Yes

### OS Platform and Distribution

Windows 11 installed with WSL Ubuntu

### Mobile device

_No response_

### Python version

python=3.9 and 3.11.3

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

cudatoolkit=11.8.0, nvidia-cudnn-cu11==8.6.0.163

### GPU model and memory

NVIDIA GeForce RTX 3070 Laptop GPU and 32GB memory

### Current Behaviour?

A bug happened!

I have no issue running the same code under windows environment (IDE: Jupyter Notebook).
However, I have problem to run the same code in WSL environment (IDE: Jupyter Notebook).

The code below is the root cause of the error message. (Note: train_set and validate_set is the output from imagedatagenerator flow_from_directory)

history = model.fit(train_set, validation_data = validate_set, epochs = 10, verbose = 2)

Error message: 
2023-05-09 01:04:47.558215: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int32
	 [[{{node Placeholder/_0}}]]

### Standalone code to reproduce the issue

```shell
import os
import csv
import time
import numpy as np
import pandas as pd
import tensorflow as tf
import matplotlib.pyplot as plt

from keras import regularizers
from keras.models import Sequential, load_model
from keras.layers import Dense, Conv2D, MaxPooling2D, Flatten, Dropout
from keras.callbacks import EarlyStopping
from keras.preprocessing.image import ImageDataGenerator
from sklearn.model_selection import GridSearchCV
from scikeras.wrappers import KerasClassifier

# Get list of physical GPU devices
gpu_list = tf.config.list_physical_devices('GPU')

if len(gpu_list) > 0:
    # Check Number of GPUs
    print('number of GPUs available:', len(gpu_list))
    print('\nGPU name:')

    # Check GPU Name
    for i in range(len(gpu_list)):
        print(str(i + 1) + '.', gpu_list[i].name.split(':', 1)[1])

    # Set memory growth for the GPU
    tf.config.experimental.set_memory_growth(gpu_list[0], True)

    # Set visible devices to only use the first GPU
    tf.config.experimental.set_visible_devices(gpu_list[0], 'GPU')

    # Verify that the GPU is being used
    print('\nUsing GPU:', gpu_list[0])

# Set attribute variable
attr = 'gender'

# Set directories
root_dir = '/mnt/c/Users/Ang/Desktop/11 DL/Assignment/DeepFashion/images'
data_dir = os.path.join(root_dir, 'data')
label_path = os.path.join(data_dir, attr + ' label.csv')
model_path = os.path.join(data_dir, attr + ' model.h5')

# Set data augmentation for train set
train_generator = ImageDataGenerator(
    rescale = 1./255,         # Normalize the data
    rotation_range = 0,       # Randomly rotate images by up to certain degrees
    width_shift_range = 0,    # Randomly shift images horizontally by up to certain percentage of the width
    height_shift_range = 0,   # Randomly shift images vertically by up to certain percentage of the height
    shear_range = 0,          # Randomly apply shear transformation with a max shear of certain percentage
    zoom_range = 0,           # Randomly zoom in/out of images by up to certain percentage
    horizontal_flip = True,   # Randomly flip images horizontally
    vertical_flip = False,    # Do not randomly flip images vertically
    fill_mode = 'nearest'     # Fill any newly created pixels with the nearest pixel value
)

# Data augmentation not applicable to validate and test set
validate_test_generator = ImageDataGenerator(rescale = 1./255)

# Set directories
train_dir = os.path.join(data_dir, 'train', attr)
validate_dir = os.path.join(data_dir, 'validate', attr)
test_dir = os.path.join(data_dir, 'test', attr)

# Set variables
target_size = (110, 75)
batch_size = 10

# Import and generate the image data for train set
train_set = train_generator.flow_from_directory(
    train_dir, 
    target_size = target_size, 
    color_mode = 'rgb', 
    class_mode = 'categorical',
    batch_size = batch_size,
    shuffle = True, 
    seed = 0
)

# Import and generate the image data for validate set
validate_set = validate_test_generator.flow_from_directory(
    validate_dir, 
    target_size = target_size, 
    color_mode = 'rgb', 
    class_mode = 'categorical', 
    batch_size = batch_size, 
    shuffle = True, 
    seed = 0
)

# Import and generate the image data for test set
test_set = validate_test_generator.flow_from_directory(
    test_dir, 
    target_size = target_size, 
    color_mode = 'rgb', 
    class_mode = 'categorical', 
    batch_size = batch_size, 
    shuffle = True, 
    seed = 0
)

# Save the label code to csv file
label_dict = train_set.class_indices

with open(label_path, 'w', newline = '') as csv_file:
    writer = csv.writer(csv_file)
    writer.writerow(['code', 'label'])

    for label, code in label_dict.items():
        writer.writerow([code, label])
        
# Get the classes for all sets
train_classes = train_set.num_classes
validate_classes = validate_set.num_classes
test_classes = test_set.num_classes

# Get the shape for all sets
train_shape = train_set.image_shape
validate_shape = validate_set.image_shape
test_shape = test_set.image_shape

# Print the classes and shape for all sets
print()
print(label_dict)
print()
print('train classes:', train_classes)
print('validate classes:', validate_classes)  
print('test classes:', test_classes)
print()
print('train shape:', train_shape)
print('validate shape:', validate_shape)
print('test shape:', test_shape)

# Set variables
c1 = 16
c2 = 32
h1 = 64
activation = 'relu'

# Build model
model = Sequential()
model.add(Conv2D(c1, (3, 3), input_shape = train_shape, 
                 padding = 'same', activation = activation))
model.add(Conv2D(c2, (3, 3), input_shape = train_shape, 
                 padding = 'same', activation = activation))
model.add(MaxPooling2D((2, 2)))
model.add(Flatten())
model.add(Dense(h1, activation = activation))
model.add(Dense(train_classes, activation = 'softmax'))

# Compile the model
model.compile(optimizer = 'adam', loss = 'categorical_crossentropy', metrics = ['accuracy'])

# Show the model summary
model.summary(line_length = 80)

# Train the model
print('\nModel Training:')
history = model.fit(train_set, validation_data = validate_set, epochs = 10, verbose = 2)
```


### Relevant log output

```shell
2023-05-09 01:04:47.558215: I tensorflow/core/common_runtime/executor.cc:1210] [/device:CPU:0] (DEBUG INFO) Executor start aborting (this does not indicate an error and you can ignore this message): INVALID_ARGUMENT: You must feed a value for placeholder tensor 'Placeholder/_0' with dtype int32
	 [[{{node Placeholder/_0}}]]
```
</details>