Why transformers with causal mask perform better without mask when overfitting training data?

I'm training a transformer model without a causal mask on the WikiText-2 Dataset to understand how a transformer would use future tokens in predicting the next token. However, based on my tests, a transformer without a causal mask is performing worse than one with a causal mask. Intuitively, this shouldn't be the case because the model has access to future tokens (and the next token itself) that should have statistical significance in predicting the next token.

For these results, I …

causal data dataset deeplearning future however next overfitting tests token tokens training training data transformer transformer model transformers