The Information of Large Language Model Geometry

This paper investigates the information encoded in the embeddings of large language models (LLMs). We conduct simulations to analyze the representation entropy and discover a power law relationship with model sizes. Building upon this observation, we propose a theory based on (conditional) entropy to elucidate the scaling law phenomenon. Furthermore, we delve into the auto-regressive structure of LLMs and examine the relationship between the last token and previous context tokens using information theory and regression techniques. Specifically, we establish a …

analyze building cs.ai cs.cl cs.it cs.lg embeddings entropy geometry information language language model language models large language large language model large language models law llms math.it observation paper power relationship representation scaling scaling law simulations the information theory