Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

arXiv:2403.09472v1 Announce Type: new
Abstract: Current AI alignment methodologies rely on human-provided demonstrations or judgments, and the learned capabilities of AI systems would be upper-bounded by human capabilities as a result. This raises a challenging research question: How can we keep improving the systems when their capabilities have surpassed the levels of humans? This paper answers this question in the context of tackling hard reasoning tasks (e.g., level 4-5 MATH problems) via learning from human annotations on easier tasks (e.g., …

abstract ai alignment ai systems alignment arxiv beyond capabilities cs.cl cs.lg current easy human humans question raises research scalable supervision systems type