Automatic Histograms: Leveraging Language Models for Text Dataset Exploration

arXiv:2402.14880v1 Announce Type: new
Abstract: Making sense of unstructured text datasets is perennially difficult, yet increasingly relevant with Large Language Models. Data workers often rely on dataset summaries, especially distributions of various derived features. Some features, like toxicity or topics, are relevant to many datasets, but many interesting features are domain specific: instruments and genres for a music dataset, or diseases and symptoms for a medical dataset. Accordingly, data workers often run custom analyses for each dataset, which is cumbersome …

abstract arxiv cs.ai cs.cl cs.hc data dataset datasets domain exploration features histograms language language models large language large language models making sense text topics toxicity type unstructured workers