Continual Vision-Language Representaion Learning with Off-Diagonal Information. (arXiv:2305.07437v1 [cs.LG])

This paper discusses the feasibility of continuously training the CLIP model
through streaming data. Then, by tracking the directional changes of the
representation vectors in the continuously updated CLIP model, we explore and
summarize these spatial variations as Spatial Disorder (SD), which can be
divided into Intra-modal Rotation and Inter-modal Deviation. Moreover, we
demonstrate how intra-modal rotation and inter-modal deviation lead to a
performance decline for CLIP on cross-modal retrieval tasks in both empirically
and theoretically. To alleviate the spatial …

arxiv clip continual data deviation information language paper representation rotation streaming streaming data through tracking training vectors vision