Building RedPajama // Vipul Ved Prakash // LLMs in Production Conference Part 2

// Abstract
Creating a new LLM is a difficult and expensive process, and there are several aspects that we need to get right — (1) a broad training dataset (2) a strong base model, (3) a well-aligned instruction dataset, (4) a carefully designed moderation subsystem, and (5) cost-effective training infrastructure coupled with an efficient software stack. Together’s central thesis is that these processes can be open-sourced, and we can harness the power of community to build and improve models, in …

abstract building conference dataset llm llms moderation part process production redpajama training vipul ved prakash