Exploring Vision Transformers for 3D Human Motion-Language Models with Motion Patches

arXiv:2405.04771v1 Announce Type: new
Abstract: To build a cross-modal latent space between 3D human motion and language, acquiring large-scale and high-quality human motion data is crucial. However, unlike the abundance of image data, the scarcity of motion data has limited the performance of existing motion-language models. To counter this, we introduce "motion patches", a new representation of motion sequences, and propose using Vision Transformers (ViT) as motion encoders via transfer learning, aiming to extract useful knowledge from the image domain …

abstract arxiv build counter cs.cv data however human image image data language language models modal performance quality scale space transformers type vision vision transformers