LLM as Dataset Analyst: Subpopulation Structure Discovery with Large Language Model

arXiv:2405.02363v1 Announce Type: new
Abstract: The distribution of subpopulations is an important property hidden within a dataset. Uncovering and analyzing the subpopulation distribution within datasets provides a comprehensive understanding of the datasets, standing as a powerful tool beneficial to various downstream tasks, including Dataset Subpopulation Organization, Subpopulation Shift, and Slice Discovery. Despite its importance, there has been no work that systematically explores the subpopulation distribution of datasets to our knowledge. To address the limitation and solve all the mentioned tasks …

abstract analyst arxiv cs.cl cs.cv dataset datasets discovery distribution hidden language language model large language large language model llm organization property shift tasks tool type understanding