Multi-Sense Language Modelling. (arXiv:2012.05776v3 [cs.CL] UPDATED)

The effectiveness of a language model is influenced by its token
representations, which must encode contextual information and handle the same
word form having a plurality of meanings (polysemy). Currently, none of the
common language modelling architectures explicitly model polysemy. We propose a
language model which not only predicts the next word, but also its sense in
context. We argue that this higher prediction granularity may be useful for end
tasks such as assistive writing, and allow for more a …

arxiv language modelling sense