Exploiting Word Semantics to Enrich Character Representations of Chinese Pre-trained Models. (arXiv:2207.05928v1 [cs.CL])

Most of the Chinese pre-trained models adopt characters as basic units for
downstream tasks. However, these models ignore the information carried by words
and thus lead to the loss of some important semantics. In this paper, we
propose a new method to exploit word structure and integrate lexical semantics
into character representations of pre-trained models. Specifically, we project
a word's embedding into its internal characters' embeddings according to the
similarity weight. To strengthen the word boundary information, we mix the …

arxiv pre-trained models semantics