‘Val_loss’ is not generated and training is getting interrupted before the completion of the last epoch for a multi-input multi-output model. This is happening when my checkpoint directory is checkpoint_dir + "/checkpoint-{epoch:02d}.tf"
This is the [error](https://github.com/keras-team/keras/issues/19339#issue-2196808063) that we see when the checkpoint directory is checkpoint_dir + "/checkpoint-{epoch:02d}-{val_loss:.2f}.tf",  @JuliaUReddy Thank you for raising this issue. 
If the issue is the same as [this](https://github.com/keras-team/keras/issues/19339) one then we will track the issue in Keras repo. Meanwhile please ensure your checkpoint directory has enough space to store all checkpoints. If training is unexpectedly interrupted, the last checkpoint might be corrupted. Try deleting existing checkpoints and restarting training. Also try with the latest TF version?
Thank you! @sushreebarsa its not the same issue as the issue #19339. Yes tried couple of times to by clearing out space in checkpoint directory but end up with the same issue.The main issue is that val_loss is not generated for all the epochs after the 1st epoch because of which there is an interruption. Yes tried with latest version of TF as well. Getting the same issue. @JuliaUReddy You've set steps_per_epoch = 213 for training, indicating it expects 213 batches per epoch. Here the training gets interrupted due to "Your input ran out of data" warnings. This happens in both the first and ninth epoch. If your dataset is finite data then, use the repeat() method while building your dataset as follows for streaming continuous data;
```
dataset = your_dataset.repeat()
```
Please let us know if that works?
Thank you! This issue is stale because it has been open for 7 days with no activity. It will be closed if no further activity occurs. Thank you. This issue was closed because it has been inactive for 7 days since being marked as stale. Please reopen if you'd like to work on this further. Are you satisfied with the resolution of your issue?
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=Yes&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/64035">Yes</a>
<a href="https://docs.google.com/forms/d/e/1FAIpQLSfaP12TRhd9xSxjXZjcZFNXPGk4kc1-qMdv3gc6bEP90vY1ew/viewform?entry.85265664=No&entry.2137816233=https://github.com/tensorflow/tensorflow/issues/64035">No</a>

### Issue type

Bug

### Have you reproduced the bug with TensorFlow Nightly?

No

### Source

source

### TensorFlow version

tf 2.12.0

### Custom code

Yes

### OS platform and distribution

MacOS Sonoma 14.1.2

### Mobile device

_No response_

### Python version

3.11.5

### Bazel version

_No response_

### GCC/compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current behavior?

I am trying to train a multi-input multi-output tensorflow model but the training gets interrupted(along with a warning) before the completion of the last epoch. Also after the 1st epoch, the val_loss is not getting generated for the rest of the following epochs. 
This behavior is seen only when I give the checkpoint directory as checkpoint_dir + "/checkpoint-{epoch:02d}.tf" and if I give the directory as checkpoint_dir + "/checkpoint-{epoch:02d}-{val_loss:.2f}.tf", I get another another error mentioned here. 
Both training and validation data are a list of numpy arrays and I am passing the validation data while training.


### Standalone code to reproduce the issue

```shell
def create_dataMLP(dim, regress=False):
 # define our MLP network
 model = Sequential()
 model.add(Dense(16, input_dim=dim, activation="relu")) 
 model.add(Dense(8, activation="relu"))   
 # check to see if the regression node should be added
 if regress:
   model.add(Dense(1, activation="linear"))
 # return our model
 return model

Function to create ImageProcessing Branch imageCNN
def create_imageCNN(name):
 model = tf.keras.Sequential([
   # Explicitly define the input shape so the model can be properly
   # loaded by the TFLiteConverter
   tf.keras.layers.InputLayer(input_shape=IMAGE_SIZE + (3,)), 
   hub.KerasLayer(model_handle, trainable=do_fine_tuning, name=name),
 ])
 return model

Function to create the multi-input and multi-out Model
def create_MultiInputMultiOutputRegressionModel():
 # create the MLP and CNN models
 #first create the dataMLP model
 model_dataMLP = create_dataMLP(data_dimension)
 print("DATA MLP model branch")
 model_dataMLP.summary()
 #next create the Neck imageCNN model
 print("\n\n----------------------")
 neck_model_imageCNN = create_imageCNN("Neck1")
 print("IMAGE CNN model branch")
 neck_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the SHOULDER imageCNN model
 print("\n\n----------------------")
 shoulder_model_imageCNN = create_imageCNN("Shoulder1")
 print("IMAGE CNN model branch")
 shoulder_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the CHEST imageCNN model
 print("\n\n----------------------")
 chest_model_imageCNN = create_imageCNN("Chest1")
 print("IMAGE CNN model branch")
 chest_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the WAIST imageCNN model
 print("\n\n----------------------")
 waist_model_imageCNN = create_imageCNN("Waist1")
 print("IMAGE CNN model branch")
 waist_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the HIPS imageCNN model
 print("\n\n----------------------")
 hips_model_imageCNN = create_imageCNN("Hips1")
 print("IMAGE CNN model branch")
 hips_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the LEGS imageCNN model
 print("\n\n----------------------")
 legs_model_imageCNN = create_imageCNN("Legs1")
 print("IMAGE CNN model branch")
 legs_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the UpperArmLeft_Breadth imageCNN model
 print("\n\n----------------------")
 Uparm_left_model_imageCNN = create_imageCNN("UpArmLeft1")
 print("IMAGE CNN model branch")
 Uparm_left_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the UpperArmRight_Breadth imageCNN model
 print("\n\n----------------------")
 Uparm_right_model_imageCNN = create_imageCNN("UpArmRight1")
 print("IMAGE CNN model branch")
 Uparm_right_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the ARM_Left_Length imageCNN model
 print("\n\n----------------------")
 arm_left_length_model_imageCNN = create_imageCNN("ArmLeft1")
 print("IMAGE CNN model branch")
 arm_left_length_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 #next create the ARM_Right_Length imageCNN model
 print("\n\n----------------------")
 arm_right_length_model_imageCNN = create_imageCNN("ArmRight1")
 print("IMAGE CNN model branch")
 arm_right_length_model_imageCNN.summary()
 print("\n\n\n\n----------------------")
 # create the input to our final set of layers as the *output* of both
 # the mode_dataMLP and ALL THE multiple model_imageCNNs
 combinedInput = concatenate([model_dataMLP.output, neck_model_imageCNN.output, shoulder_model_imageCNN.output, chest_model_imageCNN.output, waist_model_imageCNN.output, hips_model_imageCNN.output, legs_model_imageCNN.output, Uparm_left_model_imageCNN.output, Uparm_right_model_imageCNN.output, arm_left_length_model_imageCNN.output, arm_right_length_model_imageCNN.output])


 #CREATE the fully connected network after the concatenation
 x = Dense(16, activation="relu")(combinedInput) # was 8
 x =tf.keras.layers.Dropout(rate=0.2)(x)
 x = Dense(output_dimension, activation="linear", name="pred")(x)
 model = Model(inputs=[model_dataMLP.input, neck_model_imageCNN.input, shoulder_model_imageCNN.input, chest_model_imageCNN.input, waist_model_imageCNN.input, hips_model_imageCNN.input, legs_model_imageCNN.input, Uparm_left_model_imageCNN.input, Uparm_right_model_imageCNN.input, arm_left_length_model_imageCNN.input, arm_right_length_model_imageCNN.input], outputs=x)
 print("COMPLETE Multi-Input Multi-Output Model")
 model.summary()
 return model

Model compile and setting up checkpoint directory
def compileSetupModel(model, checkpoint_dir = "/content/drive/MyDrive/ObjectDetection/SmartFashion/Regression/TrainingRun/Run4"):
 #from tensorflow.keras.losses import MeanAbsoluteError, MeanAbsolutePercentageErr
 print("Contents of trainingRun directory ")
 !ls $checkpoint_dir
 #setup the  loss as MeanAbsoluteError() so will do regression
 model.compile(
   optimizer=tf.keras.optimizers.SGD(learning_rate=0.0005, momentum=0.9),
   loss = tf.keras.losses.MeanAbsoluteError())
checkpoint_dir = checkpoint_dir + "/checkpoint-{epoch:02d}.tf"
mc = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_dir, save_best_only=True, verbose=1, save_freq='epoch')
#callback for tensorboard
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)
 return mc, tensorboard_callback
mc, tensorboard_callback = compileSetupModel(model)

Model fit 
checkpoint_dir = "/content/drive/MyDrive/ObjectDetection/SmartFashion/Regression/TrainingRun/Run1"
epochs = 10  #should be more like 1000
train_size = len(trainImagesNDData)
valid_size = len(validationImagesNDData)
steps_per_epoch = (train_size // BATCH_SIZE) - 1
validation_steps = (valid_size // BATCH_SIZE) - 1
hist = model.fit(
   x=[trainNumericData, trainImagesNDData, trainImagesSBData, trainImagesCBData, trainImagesWBData, trainImagesHBData,trainImagesLLBData, trainImagesUpLeftABData,trainImagesUpRightABData, trainImagesALeftLData, trainImagesARightLData],   #x=trainImages,    # images
   y=trainAllRegressionData,  #y=train["severity"],  # severity target regression value
   epochs=epochs, steps_per_epoch=steps_per_epoch,
   validation_data=([validationNumericData, validationImagesNDData, validationImagesSBData, validationImagesCBData, validationImagesWBData, validationImagesHBData, validationImagesLLBData, validationImagesUpLeftABData, validationImagesUpRightABData, validationImagesALeftLData, validationImagesARightLData], validationAllRegressionData),
   validation_steps=validation_steps,
   callbacks=[mc, tensorboard_callback]).history   #callback for both checkpoints and tensorboard.
```


### Relevant log output

```shell
steps_per_epoch =213
validation_steps =25
Steps per epoch for training is 213
[INFO] training model...
Epoch 1/10
213/213 [==============================] - ETA: 0s - loss: 2.6573WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 25 batches). You may need to use the repeat() function when building your dataset.
Epoch 1: val_loss improved from inf to 2.23204, saving model to /content/drive/MyDrive/ObjectDetection/SmartFashion/Regression/TrainingRun/Run4/checkpoint-01.tf
213/213 [==============================] - 654s 1s/step - loss: 2.6573 - val_loss: 2.2320
Epoch 2/10
213/213 [==============================] - ETA: 0s - loss: 2.1445WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 2.1445
Epoch 3/10
213/213 [==============================] - ETA: 0s - loss: 1.8726WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 1.8726
Epoch 4/10
213/213 [==============================] - ETA: 0s - loss: 1.7671WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 542ms/step - loss: 1.7671
Epoch 5/10
213/213 [==============================] - ETA: 0s - loss: 1.6374WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 1.6374
Epoch 6/10
213/213 [==============================] - ETA: 0s - loss: 1.5755WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 1.5755
Epoch 7/10
213/213 [==============================] - ETA: 0s - loss: 1.5525WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 1.5525
Epoch 8/10
213/213 [==============================] - ETA: 0s - loss: 1.5087WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 115s 541ms/step - loss: 1.5087
Epoch 9/10
 16/213 [=>............................] - ETA: 1:22 - loss: 1.6039WARNING:tensorflow:Your input ran out of data; interrupting training. Make sure that your dataset or generator can generate at least `steps_per_epoch * epochs` batches (in this case, 2130 batches). You may need to use the repeat() function when building your dataset.
WARNING:tensorflow:Can save best model only with val_loss available, skipping.
213/213 [==============================] - 34s 156ms/step - loss: 1.6039
```
