Auxiliary task demands mask the capabilities of smaller language models

arXiv:2404.02418v1 Announce Type: new
Abstract: Developmental psychologists have argued about when cognitive capacities such as language understanding or theory of mind emerge. These debates often hinge on the concept of "task demands" -- the auxiliary challenges associated with performing a particular evaluation -- that may mask the child's underlying ability. The same issues arise when measuring the capacities of language models (LMs): performance on a task is a function of the model's underlying competence, combined with the model's ability to …

abstract arxiv capabilities challenges child cognitive concept cs.ai cs.cl evaluation hinge language language models language understanding mind theory theory of mind type understanding