FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing

arXiv:2402.13533v1 Announce Type: new
Abstract: Large language models (LLMs) are computationally intensive. The computation workload and the memory footprint grow quadratically with the dimension (layer width). Most of LLMs' parameters come from the linear layers of the transformer structure and are highly redundant. These linear layers contribute more than 80% of the computation workload and 99% of the model size. To pretrain and finetune LLMs efficiently, there are three major challenges to address: 1) reducing redundancy of the linear layers; …

abstract applications arxiv computation computing cs.ai cs.cl cs.dc cs.lg financial financial applications finetuning fingpt hpc language language models large language large language models layer linear llms memory parameters performance pretraining transformer type