A controlled study of humans vs AI (GPT-4). We have the lead, for now!

Recently stumbled upon a [paper from Durham University](https://arxiv.org/pdf/2403.16977.pdf) that pitted physics students against GPT-3.5 and GPT-4 in a university-level coding assignment.


I really liked the study because unlike benchmarks which can be fuzzy or misleading, this was a good, controlled, case study of humans vs AI on a specific task.

At a high level here were the main takeaways:

\- Students outperformed the AI models, scoring 91.9% compared to 81.1% for the best-performing AI method (GPT-4 with prompt engineering).
\- …

ai models aipromptprogramming benchmarks case case study engineering good gpt gpt-4 humans prompt scoring students study