[P] The N Implementation Details of RLHF with PPO

We are happy to share a great repro of OpenAI's early RLHF codebase, with nearly identical learning curves. We also summarized implementation details (did you know Adam Optim's implementation details could impact RLHF?)

* 📜 Blog post:https://huggingface.co/blog/the_n_implementation_details_of_rlhf_with_ppo
* 💾 Code: https://github.com/vwxyzjn/lm-human-preference-details

adam codebase impact implementation machinelearning openai ppo rlhf