Why categorical cross entropy loss is not correlated with NLP scores? I'm training a deep network for image captioning which is consist of one CNN and three GRUs. During training epoch by epoch model loss (categorical cross entropy) decreases but when I'm measuring bleu,METEOR,ROUGE,CIDEr and SPICE scores,I get best ones in the first epoch that has worst loss. I don't get why this is happening? And if categorical cross entropy is not a suitable loss function for autoencoder then what should I use instead?
There may be no mathematical linkage between categorical cross entropy and BLEU. BLEU if i remember correctly is probably a distance based measure of computer output to human judgement - there's no loss variable anywhere. This is unlike the perplexity score which IS derivable from the categorical cross entropy.
Loss function really depends on what you're comparing against your labels. Categorical cross entropy gives a way to compare the distributions of the prediction and truth labels and calculate an error based on the 'distance' between the distributions. Categorical cross entropy really works well to model the distributions of discrete vector labels, common in NLP.