[R] Understanding Dataset Difficulty with V-Usable Information (ICML 2022, oral)

Web: https://www.reddit.com/r/MachineLearning/comments/vd9xha/r_understanding_dataset_difficulty_with_vusable/

Link: https://arxiv.org/abs/2110.08420

Abstract: Estimating the difficulty of a dataset typically involves comparing state-of-the-art models to humans; the bigger the performance gap, the harder the dataset is said to be. However, this comparison provides little understanding of how difficult each instance in a given distribution is, or what attributes make the dataset difficult for a given model. To address these questions, we frame dataset difficulty -- w.r.t. a model V -- as the lack of V-usable information (Xu et al., 2019), …

2022 dataset information machinelearning