Training loss decreases dramatically after first epoch and validation loss unstable
I am using EfficientNet-B0 as a subnet in Siamese network and contrastive loss as a loss function for an image similarity task. My dataset is quite large (27550 images for training) with 2 classes. After the first epoch, the training loss decreases dramatically while the validation loss is unstable. Can overfitting happen this early? Or is there something wrong with my data that is confusing? Here is the graph I get after training my model with 100 epochs

First, Draw the training and validation loss by setting up a lower and variable learning_rate. This might happen because of higher learning rate.
Secondly, we all knows that the model Overfits, when the training loss is way smaller than the testing loss. By using, dropout, regularization and deeper model (vgg, ResNet) you can improve it.
