[R] Seeking input: transformer modification with 25-30% improvement in validation loss across 3 datasets

Long time lurker here. Made an account just to post this.

I've been experimenting with some modifications on the transformer architecture (addition of a new standalone component).

Recently I got to something that seems to be an improvement of the validation loss by \~25-30% over vanilla decoder transformers; the task is next token prediction.

My question is if this is significant enough to dedicate more serious effort in (eg. getting more compute credits to create a bigger model, running a …

architecture datasets improvement loss machinelearning something transformer transformer architecture validation validation loss