Sept. 25, 2023, 12:26 p.m. | /u/twitch-flystewie

Data Science www.reddit.com

I’m wondering how everyone handles big data. I have 12 csvs each 90 mbs each. I’ve done some analysis imported a couple individually but of course the combine csv script I use to union them on is pretty slow.

Directly importing from sql is doable but slow as well. I’m relatively new at work but just wondering what everyone else does whether they just look at smaller subsets of their data at a time or they use dask instead of …

analysis big big data course csv data datascience jupyter notebook sql them union

Software Engineer for AI Training Data (School Specific)

@ G2i Inc | Remote

Software Engineer for AI Training Data (Python)

@ G2i Inc | Remote

Software Engineer for AI Training Data (Tier 2)

@ G2i Inc | Remote

Data Engineer

@ Lemon.io | Remote: Europe, LATAM, Canada, UK, Asia, Oceania

Artificial Intelligence – Bioinformatic Expert

@ University of Texas Medical Branch | Galveston, TX

Lead Developer (AI)

@ Cere Network | San Francisco, US