Why I Stopped Dumping DataFrames to a CSV and Why You Should Too

It’s time to say goodbye to pd.to_csv() and pd.read_csv()

Building an end-to-end data-driven pipeline is challenging and demanding. Having been there myself, the process is extremely tedious, and one may inevitably end up with numerous intermediate files. While these files are usually meant to serve as checkpoints or assist further modules in the pipeline, one might be unknowingly compromising the run-time and lifting storage requirements by not choosing the appropriate format for these intermediate …

csv dataframes pandas programming