What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models

arXiv:2404.04759v1 Announce Type: cross
Abstract: Compression techniques have been crucial in advancing machine learning by enabling efficient training and deployment of large-scale language models. However, these techniques have received limited attention in the context of low-resource language models, which are trained on even smaller amounts of data and under computational constraints, a scenario known as the "low-resource double-bind." This paper investigates the effectiveness of pruning, knowledge distillation, and quantization on an exclusively low-resourced, small-data language model, AfriBERTa. Through a battery …

abstract arxiv attention compression context cs.cl cs.lg data deployment enabling however impact language language models low machine machine learning scale small small data training type