Breaking through the learning plateaus of in-context learning in Transformer

arXiv:2309.06054v3 Announce Type: replace
Abstract: In-context learning, i.e., learning from context examples, is an impressive ability of Transformer. Training Transformers to possess this in-context learning skill is computationally intensive due to the occurrence of learning plateaus, which are periods within the training process where there is minimal or no enhancement in the model's in-context learning capability. To study the mechanism behind the learning plateaus, we conceptually seperate a component within the model's internal representation that is exclusively affected by the …

abstract arxiv breaking context context learning cs.cl cs.cv cs.lg examples in-context learning process replace skill through training transformer transformers type