What Makes Instruction Learning Hard? An Investigation and a New Challenge in a Synthetic Environment. (arXiv:2204.09148v2 [cs.CL] UPDATED)

The instruction learning paradigm -- where a model learns to perform new
tasks from task descriptions alone -- has become popular in general-purpose
model research. The capabilities of large transformer models as instruction
learners, however, remain poorly understood. We use a controlled synthetic
environment to characterize such capabilities. Specifically, we use the task of
deciding whether a given string matches a regular expression (viewed as an
instruction) to identify properties of tasks, instructions, and instances that
make instruction learning challenging. …

arxiv challenge environment investigation learning