TextGram: Towards a better domain-adaptive pretraining

arXiv:2404.18228v1 Announce Type: cross
Abstract: For green AI, it is crucial to measure and reduce the carbon footprint emitted during the training of large language models. In NLP, performing pre-training on Transformer models requires significant computational resources. This pre-training involves using a large amount of text data to gain prior knowledge for performing downstream tasks. Thus, it is important that we select the correct data in the form of domain-specific data from this vast corpus to achieve optimum results aligned …

abstract arxiv carbon carbon footprint computational cs.cl cs.lg data domain green green ai knowledge language language models large language large language models nlp pre-training pretraining prior reduce resources text training transformer transformer models type