Why categorical cross entropy loss is not correlated with NLP scores?
There may be no mathematical linkage between categorical cross entropy and BLEU. BLEU if i remember correctly is probably a distance based measure of computer output to human judgement - there's no loss variable anywhere. This is unlike the perplexity score which IS derivable from the categorical cross entropy.
Loss function really depends on what you're comparing against your labels. Categorical cross entropy gives a way to compare the distributions of the prediction and truth labels and calculate an error based on the 'distance' between the distributions. Categorical cross entropy really works well to model the distributions of discrete vector labels, common in NLP.