Transcormer: Transformer for Sentence Scoring with Sliding Language Modeling. (arXiv:2205.12986v1 [cs.CL])

Sentence scoring aims at measuring the likelihood score of a sentence and is
widely used in many natural language processing scenarios, like reranking,
which is to select the best sentence from multiple candidates. Previous works
on sentence scoring mainly adopted either causal language modeling (CLM) like
GPT or masked language modeling (MLM) like BERT, which have some limitations:
1) CLM only utilizes unidirectional information for the probability estimation
of a sentence without considering bidirectional context, which affects the
scoring quality; …

arxiv language modeling scoring transformer