Task Selection for AutoML System Evaluation. (arXiv:2208.12754v1 [cs.LG])

Our goal is to assess if AutoML system changes - i.e., to the search space or
hyperparameter optimization - will improve the final model's performance on
production tasks. However, we cannot test the changes on production tasks.
Instead, we only have access to limited descriptors about tasks that our AutoML
system previously executed, like the number of data points or features. We also
have a set of development tasks to test changes, ex., sampled from OpenML with
no usage constraints. …

arxiv automl evaluation lg