Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer. (arXiv:2210.02729v3 [cs.CL] UPDATED)

Developing neural architectures that are capable of logical reasoning has
become increasingly important for a wide range of applications (e.g., natural
language processing). Towards this grand objective, we propose a symbolic
reasoning architecture that chains many join operators together to model output
logical expressions. In particular, we demonstrate that such an ensemble of
join-chains can express a broad subset of ''tree-structured'' first-order
logical expressions, named FOET, which is particularly useful for modeling
natural languages. To endow it with differentiable learning …

arxiv attention head join multi-head multi-head attention network reasoning transformer