Analysing The Impact of Sequence Composition on Language Model Pre-Training

arXiv:2402.13991v1 Announce Type: new
Abstract: Most language model pre-training frameworks concatenate multiple documents into fixed-length sequences and use causal masking to compute the likelihood of each token given its context; this strategy is widely adopted due to its simplicity and efficiency. However, to this day, the influence of the pre-training sequence composition strategy on the generalisation properties of the model remains under-explored. In this work, we find that applying causal masking can lead to the inclusion of distracting information from …

abstract arxiv compute context cs.cl documents efficiency frameworks impact influence language language model likelihood masking multiple pre-training simplicity strategy token training type