all AI news
Multilingual Search with Subword TF-IDF. (arXiv:2209.14281v2 [cs.CL] UPDATED)
Sept. 30, 2022, 1:13 a.m. | Artit Wangperawong
cs.LG updates on arXiv.org arxiv.org
Multilingual search can be achieved with subword tokenization. The accuracy
of traditional TF-IDF approaches depend on manually curated tokenization, stop
words and stemming rules, whereas subword TF-IDF (STF-IDF) can offer higher
accuracy without such heuristics. Moreover, multilingual support can be
incorporated inherently as part of the subword tokenization model training.
XQuAD evaluation demonstrates the advantages of STF-IDF: superior information
retrieval accuracy of 85.4% for English and over 80% for 10 other languages
without any heuristics-based preprocessing. The software to reproduce …
More from arxiv.org / cs.LG updates on arXiv.org
Regularization by Texts for Latent Diffusion Inverse Solvers
1 day, 21 hours ago |
arxiv.org
When can transformers reason with abstract symbols?
1 day, 21 hours ago |
arxiv.org
Jobs in AI, ML, Big Data
Data Scientist (m/f/x/d)
@ Symanto Research GmbH & Co. KG | Spain, Germany
Enterprise Data Architect
@ Pathward | Remote
Diagnostic Imaging Information Systems (DIIS) Technologist
@ Nova Scotia Health Authority | Halifax, NS, CA, B3K 6R8
Intern Data Scientist - Residual Value Risk Management (f/m/d)
@ BMW Group | Munich, DE
Analytics Engineering Manager
@ PlayStation Global | United Kingdom, London
Junior Insight Analyst (PR&Comms)
@ Signal AI | Lisbon, Lisbon, Portugal