What Can Transformer Learn with Varying Depth? Case Studies on Sequence Learning Tasks

arXiv:2404.01601v1 Announce Type: new
Abstract: We study the capabilities of the transformer architecture with varying depth. Specifically, we designed a novel set of sequence learning tasks to systematically evaluate and comprehend how the depth of transformer affects its ability to perform memorization, reasoning, generalization, and contextual generalization. We show a transformer with only one attention layer can excel in memorization but falls short in other tasks. Then, we show that exhibiting reasoning and generalization ability requires the transformer to have …

abstract architecture arxiv capabilities case case studies cs.lg learn novel reasoning set studies study tasks transformer transformer architecture type