Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Subhabrata (Subho) Mukherjee)

Recent research has focused on enhancing the capability of smaller models through imitation learning, drawing on the outputs generated by large foundation models (LFMs). A number of issues impact the quality of these models, ranging from limited imitation signals from shallow LFM outputs; small scale homogeneous training data; and most notably a lack of rigorous evaluation resulting in overestimating the small model’s capability as they tend to learn to imitate the style, but not the reasoning process of LFMs. To …

data foundation generated gpt gpt-4 imitation learning impact progressive learning quality research scale small through traces training training data