[D] On the difference (or lack thereof) between Cross-Entropy Loss and KL-Divergence

So cross-entropy(`H(p,q)`) and KL-divergence (`KL(p||q)`) relate to each other as follows:

`H(p,q) = KL(p||q) + H(p)` and `KL(p||q) = H(p,q) - H(p)`

where `p` is the data distribution and `q` is the model distribution. When `p` is constant (as is the case in most ML problems), minimizing `H(p,q)` is equivalent to minimizing `KL(p||q)`. However, there seems to be some ambiguity about this. (One practitioner claims)[https://stats.stackexchange.com/a/409271] that there is a difference in practice, because during batch gradient descent the data distribution …

cross-entropy difference divergence entropy kl-divergence loss machinelearning