[D] Why isn't RETRO mainstream / state-of-the-art within LLMs?

In 2021, Deepmind published [Improving language models by retrieving from trillions of tokens](https://arxiv.org/abs/2112.04426) and introduced a Retrieval-Enhanced Transformer (RETRO). Whereas RAG clasically involves supplementing input tokens at inference time by injecting relevant documents into context, RETRO can access related embeddings from an external database during *both* training and inference. The goal was to decouple reasoning and knowledge: by allowing as-needed lookup, the model can be freed from having to memorize all facts within its weights and instead reallocate energy toward …

claude exception faster gemini gpt llama machinelearning major mistral part sense team