My friend has made a LM architecture that might be better than the Transformer... [P]

What he made is a LM architecture inspired by the Transformer and RWKV. Its validation loss went to 1.69 after a few minutes of training on Shakespeare character-wise, for a 0.4 million parameter model. This beats the stats that NanoGPT's GitHub page says for its small model (trained on a Macbook). He doesn't yet know how it will scale, but if this does prove to be as good as it seems what would next steps be? Just publishing on GitHub …

architecture github loss machinelearning nanogpt stats training transformer validation validation loss