Learning with Noisy Foundation Models

arXiv:2403.06869v1 Announce Type: new
Abstract: Foundation models are usually pre-trained on large-scale datasets and then adapted to downstream tasks through tuning. However, the large-scale pre-training datasets, often inaccessible or too expensive to handle, can contain label noise that may adversely affect the generalization of the model and pose unexpected risks. This paper stands out as the first work to comprehensively understand and analyze the nature of noise in pre-training datasets and then effectively mitigate its impacts on downstream tasks. Specifically, …

abstract arxiv cs.ai cs.cl cs.cv cs.lg datasets foundation however noise paper pre-training risks scale tasks through training training datasets type