Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning

arXiv:2404.12897v1 Announce Type: new
Abstract: While Large Language Models (LLMs) exhibit remarkable capabilities in zero-shot and few-shot scenarios, they often require computationally prohibitive sizes. Conversely, smaller Masked Language Models (MLMs) like BERT and RoBERTa achieve state-of-the-art results through fine-tuning but struggle with extending to few-shot and zero-shot settings due to their architectural constraints. Hence, we propose Statement-Tuning, a technique that models discriminative tasks as a set of finite statements and trains an Encoder model to discriminate between the potential statements …

abstract art arxiv bert capabilities cs.cl enabling encoder few-shot fine-tuning language language models large language large language models llms natural prompting results roberta state struggle through type via zero-shot