Feb. 8, 2022, 10:41 p.m. | /u/dorepensee

Data Science www.reddit.com

I'm a student researcher and just got access to some extremely large datasets from a state govt. (I have five years worth of data and for perspective, each year is about 11GB large). I've never worked with datasets this large before and was hoping someone could guide me to resources I could use?

How do I split the files up and ‘load’, import or select fewer variables to process?

Should I work with Python/ R? Any guidance is appreciated and …

analyze datascience software

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Lead GNSS Data Scientist

@ Lurra Systems | Melbourne

Senior Machine Learning Engineer (MLOps)

@ Promaton | Remote, Europe

Alternant Data Engineering

@ Aspire Software | Angers, FR

Senior Software Engineer, Generative AI

@ Google | Dublin, Ireland