Optimizing for efficiency/memory use with spaCy and dask when preprocessing ~30M medium-large strings

I have scraped about 30 million Reddit comments. Now I want to use them to train some classification models. But this volume of data is proving seriously challenging to work with.

My current set up is that the comment strings are stored as a `dask.Series`. At first I was using `dask` methods to clean the comments in parallel (this step involves multiple passes each using regex), then using `apply(nlp)` to convert each comment into a `spacy` `Doc` (this just uses …

dask efficiency languagetechnology medium memory spacy