When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting. (arXiv:2104.09691v5 [cs.CL] UPDATED)

In 2018, Mikolov et al. introduced the positional language model, which has
characteristics of attention-based neural machine translation models and which
achieved state-of-the-art performance on the intrinsic word analogy task.
However, the positional model is not practically fast and it has never been
evaluated on qualitative criteria or extrinsic tasks. We propose a constrained
positional model, which adapts the sparse attention mechanism from neural
machine translation to improve the speed of the positional model. We evaluate
the positional and constrained …

arxiv attention