Join-Chain Network: A Logical Reasoning View of the Multi-head Attention in Transformer. (arXiv:2210.02729v1 [cs.CL])

Developing neural architectures that are capable of logical reasoning has
become increasingly important for a wide range of applications (e.g., natural
language processing). Towards this grand objective, we first propose a symbolic
reasoning architecture that chain FOET, which is particularly useful for
modeling natural languages. To endow it with differentiable learning
capability, we closely examine various neural operators for approximating the
symbolic join-chains. Interestingly, we find that the widely used multi-head
self-attention module in transformer can be understood as a …

arxiv attention head join multi-head multi-head attention network reasoning transformer