CWTM: Leveraging Contextualized Word Embeddings from BERT for Neural Topic Modeling

arXiv:2305.09329v3 Announce Type: replace
Abstract: Most existing topic models rely on bag-of-words (BOW) representation, which limits their ability to capture word order information and leads to challenges with out-of-vocabulary (OOV) words in new documents. Contextualized word embeddings, however, show superiority in word sense disambiguation and effectively address the OOV issue. In this work, we introduce a novel neural topic model called the Contextlized Word Topic Model (CWTM), which integrates contextualized word embeddings from BERT. The model is capable of learning …

abstract arxiv bag bert challenges cs.ai cs.cl documents embeddings however information leads modeling representation sense show topic modeling type word word embeddings words