Understanding the Learning Dynamics of Alignment with Human Feedback

arXiv:2403.18742v1 Announce Type: new
Abstract: Aligning large language models (LLMs) with human intentions has become a critical task for safely deploying models in real-world systems. While existing alignment approaches have seen empirical success, theoretically understanding how these methods affect model behavior remains an open question. Our work provides an initial attempt to theoretically analyze the learning dynamics of human preference alignment. We formally show how the distribution of preference datasets influences the rate of model updates and provide rigorous guarantees …

abstract alignment arxiv become behavior cs.ai cs.lg dynamics feedback human human feedback language language models large language large language models llms model behavior question success systems type understanding work world