PACT: Perception-Action Causal Transformer for Autoregressive Robotics Pre-Training. (arXiv:2209.11133v2 [cs.RO] UPDATED)

Robotics has long been a field riddled with complex systems architectures
whose modules and connections, whether traditional or learning-based, require
significant human expertise and prior knowledge. Inspired by large pre-trained
language models, this work introduces a paradigm for pre-training a general
purpose representation that can serve as a starting point for multiple tasks on
a given robot. We present the Perception-Action Causal Transformer (PACT), a
generative transformer-based architecture that aims to build representations
directly from robot data in a self-supervised …

arxiv perception pre-training robotics training transformer