classify observations into two possible classes (often simply labelled and ). The output of the model for a given observation, given a vector of input features , can be interpreted as a probability, which serves as the basis for classifying the observation. In logistic regression, the probability is modeled using the logistic function where is some function of the input vector , commonly just a linear function. The probability of the output is given by where the vector of weights is optimized through some appropriate algorithm such as gradient descent. Similarly, the complementary probability of finding the output is simply given by Having set up our notation, and , we can use cross-entropy to get a measure of dissimilarity between and : Logistic regression typically optimizes the log loss for all the observations on which it is trained, which is the same as optimizing the average cross-entropy in the sample. For example, suppose we have samples with each sample indexed by . The average of the loss function is then given by: where , with the logistic function as before. The logistic loss is sometimes called cross-entropy loss. It is also known as log loss (In this case, the binary label is often denoted by {−1,+1}). Remark: The gradient of the cross-entropy loss for logistic regression is the same as the gradient of the squared error loss for linear regression. That is, define Then we have the result The proof is as follows. For any , we have In a similar way, we eventually obtain the desired result. See also Cross-entropy method Logistic regression Conditional entropy Maximum likelihood estimation Mutual information References External links Cross Entropy Entropy and information Loss functions