HLAT: High-quality Large Language Model Pre-trained on AWS Trainium

arXiv:2404.10630v1 Announce Type: cross
Abstract: Getting large language models (LLMs) to perform well on the downstream tasks requires pre-training over trillions of tokens. This typically demands a large number of powerful computational devices in addition to a stable distributed training framework to accelerate the training. The growing number of applications leveraging AI/ML had led to a scarcity of the expensive conventional accelerators (such as GPUs), which begs the need for the alternative specialized-accelerators that are scalable and cost-efficient. AWS Trainium …

abstract applications arxiv aws aws trainium computational cs.cl cs.lg devices distributed framework language language model language models large language large language model large language models llms pre-training quality tasks tokens training trainium type