Exploiting Inductive Bias in Transformers for Unsupervised Disentanglement of Syntax and Semantics with VAEs. (arXiv:2205.05943v1 [cs.CL])

Web: http://arxiv.org/abs/2205.05943

We propose a generative model for text generation, which exhibits
disentangled latent representations of syntax and semantics. Contrary to
previous work, this model does not need syntactic information such as
constituency parses, or semantic information such as paraphrase pairs. Our
model relies solely on the inductive bias found in attention-based
architectures such as Transformers.

In the attention of Transformers, keys handle information selection while
values specify what information is conveyed. Our model, dubbed QKVAE, uses
Attention in its decoder to …

arxiv bias semantics transformers unsupervised