Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

arXiv:2404.05405v1 Announce Type: cross
Abstract: Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even …

abstract arxiv benchmarks capabilities capability capacity cs.ai cs.cl cs.lg focus knowledge language language models laws loss part physics prior relationship scaling stores studies tuples type via