Symbolic Autoencoding for Self-Supervised Sequence Learning

arXiv:2402.10575v1 Announce Type: new
Abstract: Traditional language models, adept at next-token prediction in text sequences, often struggle with transduction tasks between distinct symbolic systems, particularly when parallel data is scarce. Addressing this issue, we introduce \textit{symbolic autoencoding} ($\Sigma$AE), a self-supervised framework that harnesses the power of abundant unparallel data alongside limited parallel data. $\Sigma$AE connects two generative models via a discrete bottleneck layer and is optimized end-to-end by minimizing reconstruction loss (simultaneously with supervised loss for the parallel data), such …

abstract adept arxiv cs.ai cs.lg data framework issue language language models next power prediction struggle systems tasks text token type