How many Labelled Examples do you need for a BERT-sized Model to Beat GPT-4 on Predictive Tasks?

How does in-context learning compare to supervised approaches on predictive tasks? How many labelled examples do you need on different problems before a BERT-sized model can beat GPT-4 in accuracy? The answer might surprise you: models with fewer than 1b parameters are actually very good at classic predictive NLP, while in-context learning struggles on many problem shapes.

accuracy bert context examples gpt gpt-4 in-context learning llms parameters predictive surprise talk tasks