Encoder part of transformer learns even after removing positional encoding. Thoughts?

I am trying to solve sentiment classification problem using self-attention mechanism. The architecture is simple. One self-attention head, one feedforward layer followed by an output layer.

Initial positional encoding was added. The model overfitted (will work to mitigate it). Then I got curious what would happen if I removed positional encoding.

The model still overfitted.

Any thoughts why?

Thanks.

architecture attention classification deeplearning encoder encoding head layer part positional encoding self-attention sentiment simple solve thoughts transformer will work