AutoChunk: Automated Activation Chunk for Memory-Efficient Long Sequence Inference

arXiv:2401.10652v2 Announce Type: replace-cross
Abstract: Large deep learning models have achieved impressive performance across a range of applications. However, their large memory requirements, including parameter memory and activation memory, have become a significant challenge for their practical serving. While existing methods mainly address parameter memory, the importance of activation memory has been overlooked. Especially for long input sequences, activation memory is expected to experience a significant exponential growth as the length of sequences increases. In this approach, we propose AutoChunk, …

abstract applications arxiv automated become challenge cs.dc cs.lg cs.pf deep learning importance inference memory performance practical requirements type