The Pre-trainer's toolkit: From dataset construction to model scaling

Abstract: Recent breakthroughs in machine learning rely heavily on pre-training techniques, harnessing larger datasets, models, and computational resources to create base-models for subsequent fine-tuning. In this talk, we develop a pre-training toolkit. Drawing from empirical findings, we present methodologies for dataset construction and de-risking large-scale model training. Our discussion touches on both multimodal and language modeling domains. By addressing the entire pre-training pipeline, from dataset creation to downstream evaluation, we aim to create better, more reliable models.

Bio: Samir Yitzhak …

abstract computational construction create dataset datasets fine-tuning machine machine learning model scaling pre-training resources scale scaling talk toolkit trainer training