Contrast Sets for Evaluating Language-Guided Robot Policies

arXiv:2406.13636v1 Announce Type: cross
Abstract: Robot evaluations in language-guided, real world settings are time-consuming and often sample only a small space of potential instructions across complex scenes. In this work, we introduce contrast sets for robotics as an approach to make small, but specific, perturbations to otherwise independent, identically distributed (i.i.d.) test instances. We investigate the relationship between experimenter effort to carry out an evaluation and the resulting estimated test performance as well as the insights that can be drawn …

abstract arxiv contrast cs.lg cs.ro distributed independent instances language policies potential robot robotics sample small space test type work world