AiSAQ: All-in-Storage ANNS with Product Quantization for DRAM-free Information Retrieval

arXiv:2404.06004v1 Announce Type: cross
Abstract: In approximate nearest neighbor search (ANNS) methods based on approximate proximity graphs, DiskANN achieves good recall-speed balance for large-scale datasets using both of RAM and storage. Despite it claims to save memory usage by loading compressed vectors by product quantization (PQ), its memory usage increases in proportion to the scale of datasets. In this paper, we propose All-in-Storage ANNS with Product Quantization (AiSAQ), which offloads the compressed vectors to storage. Our method achieves $\sim$10 MB …

abstract anns approximate nearest neighbor arxiv balance cs.cl cs.ds cs.ir datasets dram free good graphs information loading memory product quantization recall retrieval save scale search speed storage type usage vectors