Training a Siamese Neural Network for object similarity assessment
without going into architecture that looks reasonable but always can be updated: Weights are dominantly updated for the negative pairs. why?

In the train set where you have (approx) 10 000 times more negative samples, you are training on them and letting the network learn almost only negative pairs, but then you test it on validation and you get what you would expect. No improvement on accuracy or loss on validation set. Make sure you are more balanced (down sample train or upsample valid/test)

EDIT: given new ifo here is a couple of suggestions. 

Indication is overffiting obviously 


Early Stopping as soon as validation dataset reaches minimum STOP training. ITs overkill after that
Covariate-shift make sure that patterns are similiar in all three sets (I assume they are but just in case)
Reduce complexity, how deep is your architecture, maybe you are over doing it. If you took some random Siamese network architecture from online it may be too much (they used it for images for example where you need additional complexity)