Limits of Transformer Language Models on Algorithmic Learning

We analyze the capabilities of Transformer language models on learning discrete algorithms. To this end, we introduce two new tasks demanding the composition of several discrete sub-tasks. On both training LLaMA models from scratch and prompting on GPT-4 and Gemini we measure learning compositions of learned primitives. We observe that the compositional capabilities of state-of-the-art Transformer language models are very limited and sample-wise scale worse than relearning all sub-tasks for a new algorithmic composition. We also present a theorem in …

algorithms analyze capabilities cs.ai cs.cl cs.lg gemini gpt gpt-4 language language models llama llama models observe prompting tasks training transformer transformer language models