all AI news
Why does sklearn.Pipeline with regex outperform spacy for text preprocessing?
June 20, 2022, 4:38 a.m. | /u/synthphreak
Natural Language Processing www.reddit.com
I need help selecting between `spacy` and `sklearn` for processing a huge text corpus. I ran a test to measure the performance of each, but the results were unexpected. Moreover, because I'm new-ish to the frameworks involved, I lack confidence that my test is completely valid. I'd really appreciate some guidance.
- [Code](https://pastebin.com/dBAW1rrD)
- [Results](https://imgur.com/a/dhJmNGi)
# Background
I'm doing a project that involves preprocessing 35 million Reddit comments. This is a massive amount of text. So I'm searching …
More from www.reddit.com / Natural Language Processing
Jobs in AI, ML, Big Data
Data Scientist (m/f/x/d)
@ Symanto Research GmbH & Co. KG | Spain, Germany
Enterprise Data Architect
@ Pathward | Remote
Diagnostic Imaging Information Systems (DIIS) Technologist
@ Nova Scotia Health Authority | Halifax, NS, CA, B3K 6R8
Intern Data Scientist - Residual Value Risk Management (f/m/d)
@ BMW Group | Munich, DE
Analytics Engineering Manager
@ PlayStation Global | United Kingdom, London
Junior Insight Analyst (PR&Comms)
@ Signal AI | Lisbon, Lisbon, Portugal