of event estimated from the training set. In other words, is the probability estimate of the model that the i-th word of the text is . The sum is averaged over the words of the test. This is a Monte Carlo estimate of the true cross-entropy, where the test set is treated as samples from . Relation to maximum likelihood In classification problems we want to estimate the probability of different outcomes. Let the estimated probability of outcome be with to-be-optimized parameters and let the frequency (empirical probability) of outcome in the training set be . Given N conditionally independent samples in the training set, then the likelihood of the parameters of the model on the training set is so the log-likelihood, divided by is so that maximizing the likelihood with respect to the parameters is the same as minimizing the cross-entropy. Cross-entropy minimization Cross-entropy minimization is frequently used in optimization and rare-event probability estimation. When comparing a distribution against a fixed reference distribution , cross-entropy and KL divergence are identical up to an additive constant (since is fixed): According to the Gibbs' inequality, both take on their minimal values when , which is for KL divergence, and for cross-entropy. In the engineering literature, the principle of minimizing KL divergence (Kullback's "Principle of Minimum Discrimination Information") is often called the Principle of Minimum Cross-Entropy (MCE), or Minxent. However, as discussed in the article Kullback–Leibler divergence, sometimes the distribution is the fixed prior reference distribution, and the distribution is optimized to be as close to as possible, subject to some constraint. In this case the two minimizations are not equivalent. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency. by restating cross-entropy to be , rather than . In fact, cross-entropy is another name for relative entropy, see Cover and Thomas and Good . On the other hand, does not agree with the literature and can be misleading. Cross-entropy loss function and logistic regression Cross-entropy can be used to define a loss function in machine learning and optimization. The true probability is the true label, and the given distribution is the predicted value of the current model. This is also known as the log loss (or logarithmic loss or logistic loss); the terms "log loss" and "cross-entropy loss" are used interchangeably. More specifically, consider a binary regression model which can be used to