Sept. 25, 2023, 12:26 p.m. | /u/twitch-flystewie

Data Science www.reddit.com

I’m wondering how everyone handles big data. I have 12 csvs each 90 mbs each. I’ve done some analysis imported a couple individually but of course the combine csv script I use to union them on is pretty slow.

Directly importing from sql is doable but slow as well. I’m relatively new at work but just wondering what everyone else does whether they just look at smaller subsets of their data at a time or they use dask instead of …

analysis big big data course csv data datascience jupyter notebook sql them union

Founding AI Engineer, Agents

@ Occam AI | New York

AI Engineer Intern, Agents

@ Occam AI | US

AI Research Scientist

@ Vara | Berlin, Germany and Remote

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Lead GNSS Data Scientist

@ Lurra Systems | Melbourne