MoST: Multi-modality Scene Tokenization for Motion Prediction

arXiv:2404.19531v1 Announce Type: new
Abstract: Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this …

abstract abstraction agent arxiv cs.cv errors generate graph information perception prediction representation tokenization traffic traffic lights type vulnerable world