In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax

arXiv:2311.07811v2 Announce Type: replace
Abstract: In-context learning (ICL) is now a common method for teaching large language models (LLMs) new tasks: given labeled examples in the input context, the LLM learns to perform the task without weight updates. Do models guided via ICL infer the underlying structure of the task defined by the context, or do they rely on superficial heuristics that only generalize to identically distributed examples? We address this question using transformations tasks and an NLI task that …

abstract arxiv case context cs.cl examples in-context learning language language models large language large language models llm llms syntax tasks teaching type updates via