Fewer Truncations Improve Language Modeling

arXiv:2404.10830v1 Announce Type: new
Abstract: In large language model training, input documents are typically concatenated together and then split into sequences of equal length to avoid padding tokens. Despite its efficiency, the concatenation approach compromises data integrity -- it inevitably breaks many documents into incomplete pieces, leading to excessive truncations that hinder the model from learning to compose logically coherent and factually consistent content that is grounded on the complete context. To address the issue, we propose Best-fit Packing, a …

abstract arxiv cs.ai cs.cl cs.lg data data integrity documents efficiency equal hinder integrity language language model language model training large language large language model modeling padding split together tokens training type