Cross-Entropy, Negative Log-Likelihood, and All That Jazz

Two closely related mathematical formulations widely used in data science, and notes on their implementations in PyTorch

TL;DR

• Negative log-likelihood minimization is a proxy problem to the problem of maximum likelihood estimation.
• Cross-entropy and negative log-likelihood are closely related mathematical formulations.
• The essential part of computing the negative log-likelihood is to “sum up the correct log probabilities.”
• The PyTorch implementations of CrossEntropyLoss and NLLLoss are slightly different in the expected input values. In short, …

cross-entropy data science entropy loss-function machine learning negative pytorch