Compute efficient way to scale LLM (Journey around data/model/compute)

Compute-efficient Way to Scale LLM — Journey around data, model, and compute

Context

We have repeatedly seen that increasing the model parameters results in better performance (GPT-1 has 117M parameters, GPT-2 has 1.5B parameters, and GPT-3 has 175B parameters). But the next set of questions is how to scale the AI model. Simply increasing the model parameters without increasing the compute won’t help. There are a lot of things around a number of model parameters (N), number of compute available …

ai llm scaling