Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation

arXiv:2109.01048v3 Announce Type: replace
Abstract: Existing technologies expand BERT from different perspectives, e.g. designing different pre-training tasks, different semantic granularities, and different model architectures. Few models consider expanding BERT from different text formats. In this paper, we propose a heterogeneous knowledge language model (\textbf{HKLM}), a unified pre-trained language model (PLM) for all forms of text, including unstructured text, semi-structured text, and well-structured text. To capture the corresponding relations among these multi-format knowledge, our approach uses masked language model objective to …

abstract architectures arxiv bert cs.cl designing domain expand knowledge language language model paper perspectives pre-training representation semantic tasks technologies text training type