Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training

arXiv:2404.10555v1 Announce Type: new
Abstract: Large language models (LLMs) are now widely used in various fields, including finance. However, Japanese financial-specific LLMs have not been proposed yet. Hence, this study aims to construct a Japanese financial-specific LLM through continual pre-training. Before tuning, we constructed Japanese financial-focused datasets for continual pre-training. As a base model, we employed a Japanese LLM that achieved state-of-the-art performance on Japanese financial benchmarks among the 10-billion-class parameter models. After continual pre-training using the datasets and the …

abstract arxiv construct construction continual cs.cl datasets domain fields finance financial however japanese language language model language models large language large language model large language models llm llms pre-training q-fin.cp study through training type