Collage: Light-Weight Low-Precision Strategy for LLM Training

arXiv:2405.03637v1 Announce Type: new
Abstract: Large models training is plagued by the intense compute cost and limited hardware memory. A practical solution is low-precision representation but is troubled by loss in numerical accuracy and unstable training rendering the model less useful. We argue that low-precision floating points can perform well provided the error is properly compensated at the critical locations in the training process. We propose Collage which utilizes multi-component float representation in low-precision to accurately perform operations with numerical …

abstract accuracy arxiv compute compute cost cost cs.lg error hardware large models light llm llm training loss low memory numerical practical precision rendering representation solution strategy training type