Semi-Parametric Retrieval via Binary Token Index

arXiv:2405.01924v1 Announce Type: new
Abstract: The landscape of information retrieval has broadened from search services to a critical component in various advanced applications, where indexing efficiency, cost-effectiveness, and freshness are increasingly important yet remain less explored. To address these demands, we introduce Semi-parametric Vocabulary Disentangled Retrieval (SVDR). SVDR is a novel semi-parametric retrieval framework that supports two types of indexes: an embedding-based index for high effectiveness, akin to existing neural retrieval methods; and a binary token index that allows for …

abstract advanced applications arxiv binary cost cost-effectiveness cs.ai cs.cl cs.ir efficiency framework index indexing information landscape novel parametric retrieval search services token type via