Technical Report: Masked Skeleton Sequence Modeling for Learning Larval Zebrafish Behavior Latent Embeddings

arXiv:2403.15693v1 Announce Type: new
Abstract: In this report, we introduce a novel self-supervised learning method for extracting latent embeddings from behaviors of larval zebrafish. Drawing inspiration from Masked Modeling techniquesutilized in image processing with Masked Autoencoders (MAE) \cite{he2022masked} and in natural language processing with Generative Pre-trained Transformer (GPT) \cite{radford2018improving}, we treat behavior sequences as a blend of images and language. For the skeletal sequences of swimming zebrafish, we propose a pioneering Transformer-CNN architecture, the Sequence Spatial-Temporal Transformer (SSTFormer), designed to …

abstract arxiv autoencoders behavior cs.cv embeddings generative generative pre-trained transformer image image processing inspiration language language processing modeling natural natural language natural language processing novel processing report self-supervised learning supervised learning technical transformer type