Human Guided Exploitation of Interpretable Attention Patterns in Summarization and Topic Segmentation. (arXiv:2112.05364v2 [cs.CL] UPDATED)

The multi-head self-attention mechanism of the transformer model has been
thoroughly investigated recently. In one vein of study, researchers are
interested in understanding why and how transformers work. In another vein,
researchers propose new attention augmentation methods to make transformers
more accurate, efficient and interpretable. In this paper, we combine these two
lines of research in a human-in-the-loop pipeline to first discover important
task-specific attention patterns. Then those patterns are injected, not only to
smaller models, but also to the …

arxiv attention exploitation human patterns segmentation summarization