BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence
You just stumble over one big problem in the NLP field : finding the perfect metric..



Most traditional metrics (BLEU, ROUGE, ...) simply does not take into account the distance in terms of semantics between barking and crying.

So according to these metrics, The dog is crying is as similar as The dog is salmon to the reference, the dog is barking.
From a human viewpoint, this is not correct, the first sentence is closer to the reference, because for example the second sentence makes no sense.



People recently tried to provide better metrics in this sense. You might be interested in BERT score.

The idea is simply to use a BERT model (that have been pretrained, therefore have some linguistic knowledge) to compute how similar 2 sentences are.