BERT-LSH: Reducing Absolute Compute For Attention

arXiv:2404.08836v1 Announce Type: cross
Abstract: This study introduces a novel BERT-LSH model that incorporates Locality Sensitive Hashing (LSH) to approximate the attention mechanism in the BERT architecture. We examine the computational efficiency and performance of this model compared to a standard baseline BERT model. Our findings reveal that BERT-LSH significantly reduces computational demand for the self-attention layer while unexpectedly outperforming the baseline model in pretraining and fine-tuning tasks. These results suggest that the LSH-based attention mechanism not only offers computational …

arxiv attention bert compute cs.ai cs.cl cs.lg lsh type