Towards Interactively Improving ML Data Preparation Code via "Shadow Pipelines"

arXiv:2404.19591v1 Announce Type: cross
Abstract: Data scientists develop ML pipelines in an iterative manner: they repeatedly screen a pipeline for potential issues, debug it, and then revise and improve its code according to their findings. However, this manual process is tedious and error-prone. Therefore, we propose to support data scientists during this development cycle with automatically derived interactive suggestions for pipeline improvements. We discuss our vision to generate these suggestions with so-called shadow pipelines, hidden variants of the original pipeline …

abstract arxiv code cs.db cs.lg cs.se data data preparation data scientists debug error however improving iterative ml pipelines pipeline pipelines process scientists shadow support type via