Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models

arXiv:2403.15498v1 Announce Type: new
Abstract: Language models have shown unprecedented capabilities, sparking debate over the source of their performance. Is it merely the outcome of learning syntactic patterns and surface level statistics, or do they extract semantics and a world model from the text? Prior work by Li et al. investigated this by training a GPT model on synthetic, randomly generated Othello games and found that the model learned an internal representation of the board state. We extend this work …

abstract arxiv capabilities chess cs.cl cs.lg extract language language models patterns performance playing prior semantics statistics surface text type work world world model world models