[R] Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

**Paper**: [https://arxiv.org/abs/2404.05405](https://arxiv.org/abs/2404.05405)

**Abstract**:

>Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establish that language models can and only can store 2 bits of knowledge per parameter, even when quantized …

abstract benchmarks capabilities capability capital datasets focus knowledge language language models laws loss machinelearning multiple page prior relationship scaling stores studies through tuples usa via washington wikipedia