Dataverse: Open-Source ETL (Extract, Transform, Load) Pipeline for Large Language Models

arXiv:2403.19340v1 Announce Type: new
Abstract: To address the challenges associated with data processing at scale, we propose Dataverse, a unified open-source Extract-Transform-Load (ETL) pipeline for large language models (LLMs) with a user-friendly design at its core. Easy addition of custom processors with block-based interface in Dataverse allows users to readily and efficiently use Dataverse to build their own ETL pipeline. We hope that Dataverse will serve as a vital tool for LLM development and open source the entire library to …

abstract arxiv block challenges core cs.ai cs.cl data data processing design easy etl extract language language models large language large language models llms pipeline processing processors scale type