July 22, 2023, 6:11 a.m. | /u/Usual_Office_1740

Data Science www.reddit.com

I have a set of web scraping scripts that I'm getting ready to run. I expect the end result to produce a data set with a little over 400,000 data points. Each item will include two strings. Im concerned that my poor little laptop wont like handling such a large data set. Its an i3 with 8 gig of ram.

My three ideas for how to organize and manage this data are as follows.

1. Load them into a python …

data datascience data set gather organize scraping scripts set web web scraping

Founding AI Engineer, Agents

@ Occam AI | New York

AI Engineer Intern, Agents

@ Occam AI | US

AI Research Scientist

@ Vara | Berlin, Germany and Remote

Data Architect

@ University of Texas at Austin | Austin, TX

Data ETL Engineer

@ University of Texas at Austin | Austin, TX

Codec Avatars Research Engineer

@ Meta | Pittsburgh, PA