ClipFormer: Key-Value Clipping of Transformers on Memristive Crossbars for Write Noise Mitigation

Transformers have revolutionized various real-world applications from natural language processing to computer vision. However, traditional von-Neumann computing paradigm faces memory and bandwidth limitations in accelerating transformers owing to their massive model sizes. To this end, In-memory Computing (IMC) crossbars based on Non-volatile Memories (NVMs), due to their ability to perform highly parallelized Matrix-Vector-Multiplications (MVMs) with high energy-efficiencies, have emerged as a promising solution for accelerating transformers. However, analog MVM operations in crossbars introduce non-idealities, such as stochastic read & write …

applications bandwidth computer computer vision computing cs.et cs.lg in-memory in-memory computing key language language processing limitations massive memories memory natural natural language natural language processing noise paradigm processing transformers value vision world