Modeling Selective Feature Attention for Representation-based Siamese Text Matching

arXiv:2404.16776v1 Announce Type: new
Abstract: Representation-based Siamese networks have risen to popularity in lightweight text matching due to their low deployment and inference costs. While word-level attention mechanisms have been implemented within Siamese networks to improve performance, we propose Feature Attention (FA), a novel downstream block designed to enrich the modeling of dependencies among embedding features. Employing "squeeze-and-excitation" techniques, the FA block dynamically adjusts the emphasis on individual features, enabling the network to concentrate more on features that significantly contribute …

abstract arxiv attention attention mechanisms block costs cs.cl deployment feature inference inference costs low modeling networks novel performance representation text type while word