TF Hangs when calculating validation sample weights
In addition to this, isn't it a bug that setting the `class_weight` parameter in `model.fit` does not apply those class weights to the `validation_data`?

For example, this code:


```py
model.fit(
    X_trn, 
    y_trn,
    # NOTE: Validation loss is *incorrect* because it doesn't take into account the class weights
    validation_data=(X_val, y_val),
    class_weight=calculate_weights_for_my_imbalanced_classes(y_trn),
)
```

results in `loss` values around 0.001, but `val_loss` values of around 4.0, because I have one very dominant class (~99% of the observations) and 49 classes in the remaining 1%.

Shouldn't the `class_weight` dictionary be applied to the `val_loss` calculation if `validation_data` is provided? @sushreebarsa is this a known issue? or is there any solution planned? Hello @beyarkay! I was able to run the code successfully on colab using TF v2.12, please find the [gist](https://colab.research.google.com/gist/sushreebarsa/9aa8f3c9c8b6e7b0ecb705f364ac6d1a/60267.ipynb) here and  try to upgrade the TF version to latest. Thank you! I am unable to update to the latest version of TF because of [this macos/metal bug](https://developer.apple.com/forums/thread/722389?740707022). Is there any fix that I can use besides updating? (since it seems like apple isn't going to be fixing the issue super soon) @beyarkay Thank you for the response!
@SuryanarayanaY Could you please have a look at this issue?
Thank you! Hi @beyarkay ,

If you want to use latest tensorflow-macos with metal plugin, you need to use legacy optimizer. The constraint you have mentioned for updating to new version is due to the optimizer issue.Suppose if you are using `keras.optimizers.Adam(1e-5)` replace it with `tf.keras.optimizers.legacy.Adam(1e-5)` and it should work fine. 

Please try this and let us know of still having any problem. Thanks!
 Thanks! That works perfectly. Are there any differences between `keras.optimizers.Adam` and `tf.keras.optimizers.legacy.Adam` that I should know about which might cause me issues in the future? Okay nevermind, it's not working now for some reason. I'll paste the code I'm using (although I literally only used the legacy version of Adam) but it's having the same error. I've tried restarting my notebook, but no luck.


<details>
<summary>Code</summary>

```py
import numpy as np
import keras
import tensorflow as tf
import sklearn.model_selection

X = np.random.randint(low=300, high=900, size=(215699, 30, 30))
y = np.random.randint(low=0, high=51, size=(215699,))
X_trn, X_val, y_trn, y_val = sklearn.model_selection.train_test_split(X, y)

m = tf.keras.Sequential(
    [
        keras.layers.Input(shape=X.shape[1:]),
        keras.layers.Flatten(),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(len(np.unique(y))),
    ]
)
m.compile(
    optimizer=tf.keras.optimizers.legacy.Adam(1e-5),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)

sample_weights_val = np.ones(y_val.shape)

history = m.fit(
    X_trn,
    y_trn,
    validation_data=(X_val, y_val, sample_weights_val),
    batch_size=128,
    epochs=5,
)
```

</details>

And the output is:

```
Epoch 1/5
  12/1264 [..............................] - ETA: 5s - loss: 844.2266  

2023-04-14 11:53:32.963873: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

1256/1264 [============================>.] - ETA: 0s - loss: 125.0102
``` @beyarkay ,

I have replicated the issue with tensorflow-macos==2.11 version and its still an issue with 2.11 version also. The code hangs as reported and i have to interrupt it manually after around 2 minutes.For reference I have attached the logs below.

```
(tf-metal) suryanarayanay-macbookpro:~ suryanarayanay$ python 60267.py
Metal device set to: Apple M1 Pro

systemMemory: 16.00 GB
maxCacheSize: 5.33 GB

2023-04-14 19:14:12.631493: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:306] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support.
2023-04-14 19:14:12.631513: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:272] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: <undefined>)
2023-04-14 19:14:13.425947: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
Epoch 1/5
2023-04-14 19:14:13.569687: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:114] Plugin optimizer for device_type GPU is enabled.
1264/1264 [==============================] - ETA: 0s - loss: 111.5523^CTraceback (most recent call last):
  File "/Users/suryanarayanay/60267.py", line 35, in <module>
    history = m.fit(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 65, in error_handler
    return fn(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/training.py", line 1680, in fit
    self._eval_data_handler = data_adapter.get_data_handler(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 1579, in get_data_handler
    return DataHandler(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 1259, in __init__
    self._adapter = adapter_cls(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/data_adapter.py", line 251, in __init__
    (sample_weights, _, _) = training_utils.handle_partial_sample_weights(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/training_utils.py", line 78, in handle_partial_sample_weights
    partial_sample_weight = any_sample_weight and any(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/keras/engine/training_utils.py", line 78, in <genexpr>
    partial_sample_weight = any_sample_weight and any(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 7327, in __next__
    result = self._tensor[self._index]
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
    return fn(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler
    return dispatch_target(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py", line 1096, in _slice_helper
    return strided_slice(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
    return fn(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/util/dispatch.py", line 1176, in op_dispatch_handler
    return dispatch_target(*args, **kwargs)
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/ops/array_ops.py", line 1269, in strided_slice
    op = gen_array_ops.strided_slice(
  File "/Users/suryanarayanay/miniconda/envs/tf-metal/lib/python3.9/site-packages/tensorflow/python/ops/gen_array_ops.py", line 10672, in strided_slice
    _result = pywrap_tfe.TFE_Py_FastPathExecute(
KeyboardInterrupt

(tf-metal) suryanarayanay-macbookpro:~ suryanarayanay$ python3 -c "import tensorflow as tf; print(tf.__version__)"
2.11.0
(tf-metal) suryanarayanay-macbookpro:~ suryanarayanay$ 
```

I have also tested with tensorflow-macos 2.12 versions and the same behaviour observed there also.Attched logs below for reference.
[60267_logs(macos-2.12).txt](https://github.com/tensorflow/tensorflow/files/11233613/60267_logs.macos-2.12.txt)
 Thanks for looking at it! Is there any hope for a fix? @nitins17 , Do you have any pointers ? The same code works fine with tensorflow==2.12 on colab but fails with tensorflow-macos. 

cc: @learning-to-play 
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Performance

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

unknown 2.9.2

### Custom Code

Yes

### OS Platform and Distribution

MacOS 12.4

### Mobile device

NA

### Python version

Python 3.10.5

### Bazel version

NA

### GCC/Compiler version

NA

### CUDA/cuDNN version

NA

### GPU model and memory

NA

### Current Behaviour?

While using `validation_data` in `model.fit`, TF hangs when it's finished the first epoch (but before updating the progress bar with the validation loss). 

Specifically, this happens when a 3-tuple is passed to `validation_data`, where the third element is an array specifying the sample weights.

This happens even if the sample weights are all `1.0`.

### Standalone code to reproduce the issue

```shell
import numpy as np
import keras
import tensorflow as tf
import sklearn.model_selection

X = np.random.randint(low=300, high=900, size=(215699, 30, 30))
y = np.random.randint(low=0, high=51, size=(215699,))
X_trn, X_val, y_trn, y_val = sklearn.model_selection.train_test_split(X, y)

m = tf.keras.Sequential(
    [
        keras.layers.Input(shape=X.shape[1:]),
        keras.layers.Flatten(),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(units=64, activation="relu",),
        keras.layers.Dense(len(np.unique(y))),
    ]
)
m.compile(
    optimizer=keras.optimizers.Adam(1e-5),
    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
)

sample_weights_val = np.ones(y_val.shape)

history = m.fit(
    X_trn,
    y_trn,
    validation_data=(X_val, y_val, sample_weights_val),
    batch_size=128,
    epochs=5,
)
```


### Relevant log output

```shell
Epoch 1/5

2023-04-08 13:18:01.491389: W tensorflow/core/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz
2023-04-08 13:18:01.621051: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

1264/1264 [==============================] - ETA: 0s - loss: 124.2101
```

At this point Tensorflow stops and I've got to interrupt it in order to get control back.
</details>