all AI news
Multilingual Search with Subword TF-IDF. (arXiv:2209.14281v1 [cs.CL])
Sept. 29, 2022, 1:12 a.m. | Artit Wangperawong
cs.LG updates on arXiv.org arxiv.org
Multilingual search can be achieved with subword tokenization. The accuracy
of traditional TF-IDF approaches depend on manually curated tokenization, stop
words and stemming rules, whereas subword TF-IDF (STF-IDF) can offer higher
accuracy without such heuristics. Moreover, multilingual support can be
incorporated inherently as part of the subword tokenization model training.
XQuAD evaluation demonstrates the advantages of STF-IDF: superior information
retrieval accuracy of 85.4% for English and over 80% for 10 other languages
without any heuristics-based preprocessing. The software to reproduce …
More from arxiv.org / cs.LG updates on arXiv.org
Jobs in AI, ML, Big Data
AI Research Scientist
@ Vara | Berlin, Germany and Remote
Data Architect
@ University of Texas at Austin | Austin, TX
Data ETL Engineer
@ University of Texas at Austin | Austin, TX
Lead GNSS Data Scientist
@ Lurra Systems | Melbourne
Senior Machine Learning Engineer (MLOps)
@ Promaton | Remote, Europe
Senior Data Scientist
@ ITE Management | New York City, United States