[N] OpenLLaMA: An Open Reproduction of LLaMA

https://github.com/openlm-research/open_llama

> We train our models on the RedPajama dataset released by Together, which is a reproduction of the LLaMA training dataset containing over 1.2 trillion tokens. We follow the exactly same preprocessing steps and training hyperparameters as the original LLaMA paper, including model architecture, context length, training steps, learning rate schedule, and optimizer. The only difference between our setting and the original one is the dataset used: OpenLLaMA employs the RedPajama dataset rather than the one utilized by the …

architecture context dataset difference llama machinelearning paper rate redpajama together tokens training