Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models

arXiv:2402.19465v1 Announce Type: new
Abstract: Ensuring the trustworthiness of large language models (LLMs) is crucial. Most studies concentrate on fully pre-trained LLMs to better understand and improve LLMs' trustworthiness. In this paper, to reveal the untapped potential of pre-training, we pioneer the exploration of LLMs' trustworthiness during this period, focusing on five key dimensions: reliability, privacy, toxicity, fairness, and robustness. To begin with, we apply linear probing to LLMs. The high probing accuracy suggests that \textit{LLMs in early pre-training can …

arxiv cs.ai cs.cl dynamics language language models large language large language models pre-training tracing training type