Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks

We propose that small pretrained foundational generative language models with millions of parameters can be utilized as a general learning framework for sequence-based tasks. Our proposal overcomes the computational resource, skill set, and timeline challenges associated with training neural networks and language models from scratch. Further, our approach focuses on creating small and highly specialized models that can accurately execute a challenging task of which the base model is incapable of performing. We demonstrate that 125M, 350M, and 1.3B parameter …

challenges computational cs.ai cs.cl cs.lg framework frameworks general generative language language models networks neural networks parameters set small tasks timeline training