Understanding Performance of Long-Document Ranking Models through Comprehensive Evaluation and Leaderboarding

arXiv:2207.01262v2 Announce Type: replace-cross
Abstract: We evaluated 20+ Transformer models for ranking of long documents (including recent LongP models trained with FlashAttention) and compared them with simple FirstP baselines (applying the same model to input truncated to the first 512 tokens). We used MS MARCO Documents v1 as a primary training set and evaluated models in the zero-shot scenario as well as after fine-tuning on other collections.
In our initial experiments with standard collections we found that long-document models underperformed …

abstract arxiv cs.cl cs.ir document documents evaluation performance ranking simple them through tokens transformer transformer models type understanding