The Matrix: A Bayesian learning model for LLMs

In this paper, we introduce a Bayesian learning model to understand the behavior of Large Language Models (LLMs). We explore the optimization metric of LLMs, which is based on predicting the next token, and develop a novel model grounded in this principle. Our approach involves constructing an ideal generative text model represented by a multinomial transition probability matrix with a prior, and we examine how LLMs approximate this matrix. We discuss the continuity of the mapping between embeddings and multinomial …

bayesian behavior cs.ai cs.lg explore generative language language models large language large language models llms matrix next novel optimization paper text the matrix token