Towards Cost-Efficient Use of Pre-trained Models

Abstract: Large language models are leading to many exciting breakthroughs, but this comes at a significant cost in terms of both computational and data labeling expenses. Training state-of-the-art models requires access to high-end GPUs for pre-training and inference, in addition to labeled data for fine-tuning. In this talk I will examine the tradeoff between these costs, with the goal of supporting better decisions. Conventional wisdom holds that annotating data is expensive, so computational methods that use unlabeled data to improve …

abstract art computational cost data data labeling fine-tuning gpus inference labeling language language models large language large language models pre-trained models pre-training state talk terms training