Why is the validation performance better than the training performance? I am training a classifier to identify 24 hand signs of American Sign Language. I created a custom dataset by recording videos in different backgrounds for each of the signs and later converted the videos into images. Each sign has 3000 images, that were randomly selected to generate a training dataset with 2400 images/sign and validation dataset with the remaining 600 images/sign.


Total number of images in entire dataset: 3000 * 24 = 72000
Training dataset: 2400 * 24 = 57600
Validation dataset: 600 * 24 = 14400
Image dimension (Width x Height): 1280 x 720 pixels


The CNN architecture used for training

model = Sequential([
    Conv2D(32, (3, 3), activation='relu', input_shape=(IMG_HEIGHT, IMG_WIDTH ,3)),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(32, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Conv2D(64, (3, 3), activation='relu'),
    MaxPooling2D(pool_size=(2,2)),
    Dropout(0.25),

    Flatten(),
    Dense(128, activation='relu'),
    Dropout(0.25),

    Dense(NUM_CLASSES, activation='softmax')
])


Training parameters:

IMG_HEIGHT = 224
IMG_WIDTH = 224
BATCH_SIZE = 32
NUM_CLASSES = 24
train_datagen = ImageDataGenerator(rescale = 1./255,
                                   width_shift_range=0.1,
                                   height_shift_range=0.1,
                                   zoom_range=0.1,
                                   fill_mode='constant')
EPOCHS = 20
STEPS_PER_EPOCH = TRAIN_TOTAL // BATCH_SIZE
VALIDATION_STEPS = VALIDATION_TOTAL // BATCH_SIZE

callbacks_list = [
    tf.keras.callbacks.EarlyStopping(monitor = 'accuracy',
                                     min_delta = 0.005,
                                     patience = 3),
    tf.keras.callbacks.ModelCheckpoint(filepath = 'D:\\Models\\HSRS_ThesisDataset_5Mar_1330.h5',
                                       monitor= 'val_loss',
                                       save_best_only = True)
]

optimizer = 'adam'


The model accuracy and model loss graph is shown in the figure below:



The results obtained at the end of the training are


Train acc: 0.8000121
Val acc: 0.914441


I read this article explaining why the validation loss is lower than the training loss I want to know:


Is it because of the smaller dataset and random shuffling of the images?
Is there any way to improve the condition without changing the dataset?
Will this have a very detrimental effect on the model performance in real test cases? If not, can I just focus on improving the training accuracy of the overall model?
Assuming you pass through the entire validation dataset, this can't be due to shuffling since you still compute the loss/accuracy over the entire dataset, so order does not really matter here. It is more likely that you have a significantly smaller or less representative validation dataset, e.g., distribution of the validation dataset can be skewed towards classes where your model performs better.
What do you mean exactly by improving the situation? Having a better validation accuracy is not necessarily bad. In any case, if you decrease the effect of regularization, e.g., lowering weight decay, training accuracy might go up but your model might generalize worse, i.e.,  you might get a lower validation accuracy.
No, the goal of training is never to maximize training accuracy. You can trivially do so by just memorizing the training dataset. In short, the goal of training is to get good generalization and as long as you get a satisfactory validation accuracy, it is likely that this has happened to some degree (assuming you have a good validation dataset of course).