Prompt When the Animal is: Temporal Animal Behavior Grounding with Positional Recovery Training

arXiv:2405.05523v1 Announce Type: new
Abstract: Temporal grounding is crucial in multimodal learning, but it poses challenges when applied to animal behavior data due to the sparsity and uniform distribution of moments. To address these challenges, we propose a novel Positional Recovery Training framework (Port), which prompts the model with the start and end times of specific animal behaviors during training. Specifically, Port enhances the baseline model with a Recovering part to predict flipped label sequences and align distributions with a …

abstract arxiv behavior challenges cs.ai cs.cv data distribution framework moments multimodal multimodal learning novel prompt prompts recovery sparsity temporal training type uniform