Keypoint Action Tokens Enable In-Context Imitation Learning in Robotics

arXiv:2403.19578v1 Announce Type: cross
Abstract: We show that off-the-shelf text-based Transformers, with no additional training, can perform few-shot in-context visual imitation learning, mapping visual observations to action sequences that emulate the demonstrator's behaviour. We achieve this by transforming visual observations (inputs) and trajectories of actions (outputs) into sequences of tokens that a text-pretrained Transformer (GPT-4 Turbo) can ingest and generate, via a framework we call Keypoint Action Tokens (KAT). Despite being trained only on language, we show that these Transformers …

arxiv context cs.lg cs.ne cs.ro imitation learning robotics tokens type