Why does GPT-3 know how to answer questions and GPT-J doesn't?

When you ask GPT-3 something (through openai's online demo/playground), it will usually generate a one paragraph reply that is on topic.

Meanwhile, GPT-J will often start rambling in first person without ever directly answering the question. The same seems to happen for the smaller (6B) version of OPT.

I can speculate that this may be due to a hidden preprompts that are added to your GPT-3 input, to a difference in the training data, or simply due to the larger …

gpt gpt-3 gpt-j languagetechnology