OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework

arXiv:2404.14619v1 Announce Type: cross
Abstract: The reproducibility and transparency of large language models are crucial for advancing open research, ensuring the trustworthiness of results, and enabling investigations into data and model biases, as well as potential risks. To this end, we release OpenELM, a state-of-the-art open language model. OpenELM uses a layer-wise scaling strategy to efficiently allocate parameters within each layer of the transformer model, leading to enhanced accuracy. For example, with a parameter budget of approximately one billion parameters, …

abstract art arxiv biases cs.ai cs.cl cs.lg data enabling family framework inference investigations language language model language models large language large language models release reproducibility research results risks state training transparency type