Web: http://arxiv.org/abs/2205.05092

May 12, 2022, 1:10 a.m. | Kaitlyn Zhou, Kawin Ethayarajh, Dallas Card, Dan Jurafsky

cs.CL updates on arXiv.org arxiv.org

Cosine similarity of contextual embeddings is used in many NLP tasks (e.g.,
QA, IR, MT) and metrics (e.g., BERTScore). Here, we uncover systematic ways in
which word similarities estimated by cosine over BERT embeddings are
understated and trace this effect to training data frequency. We find that
relative to human judgements, cosine similarity underestimates the similarity
of frequent words with other instances of the same word or other words across
contexts, even after controlling for polysemy and other factors. We …

arxiv embedding words

More from arxiv.org / cs.CL updates on arXiv.org

Director, Applied Mathematics & Computational Research Division

@ Lawrence Berkeley National Lab | Berkeley, Ca

Business Data Analyst

@ MainStreet Family Care | Birmingham, AL

Assistant/Associate Professor of the Practice in Business Analytics

@ Georgetown University McDonough School of Business | Washington DC

Senior Data Science Writer

@ NannyML | Remote

Director of AI/ML Engineering

@ Armis Industries | Remote (US only), St. Louis, California

Digital Analytics Manager

@ Patagonia | Ventura, California