Augmenting Transformers with Recursively Composed Multi-grained Representations

arXiv:2309.16319v2 Announce Type: replace
Abstract: We present ReCAT, a recursive composition augmented Transformer that is able to explicitly model hierarchical syntactic structures of raw texts without relying on gold trees during both learning and inference. Existing research along this line restricts data to follow a hierarchical tree structure and thus lacks inter-span communications. To overcome the problem, we propose a novel contextual inside-outside (CIO) layer that learns contextualized representations of spans through bottom-up and top-down passes, where a bottom-up pass …

abstract arxiv communications cs.ai cs.cl data hierarchical inference line raw recursive research transformer transformers tree trees type