Larger language models do in-context learning differently

There have recently been tremendous advances in language models, partly because they can perform tasks with strong performance via in-context learning (ICL), a process whereby models are prompted with a few examples of input-label pairs before performing the task on an unseen evaluation example. In general, models’ success at in-context learning is enabled by:

• Their use of semantic prior knowledge from pre-training to predict labels while following …

context examples google google research jerry language language models machine intelligence machine learning natural language processing performance process research