Quoting Rob Lynch

gpt-4-turbo over the API produces (statistically significant) shorter completions when it "thinks" its December vs. when it thinks its May (as determined by the date in the system prompt).

I took the same exact prompt over the API (a code completion task asking to implement a machine learning task without libraries).

I created two system prompts, one that told the API it was May and another that it was December and then compared the distributions.

For the May system prompt, …

ai api code code completion generativeai gpt gpt-4 gpt4 gpt-4-turbo libraries llms machine machine learning prompt prompts rob turbo