Exploiting Optical Flow Guidance for Transformer-Based Video Inpainting

arXiv:2301.10048v2 Announce Type: replace
Abstract: Transformers have been widely used for video processing owing to the multi-head self attention (MHSA) mechanism. However, the MHSA mechanism encounters an intrinsic difficulty for video inpainting, since the features associated with the corrupted regions are degraded and incur inaccurate self attention. This problem, termed query degradation, may be mitigated by first completing optical flows and then using the flows to guide the self attention, which was verified in our previous work - flow-guided transformer …

abstract arxiv attention cs.cv features flow guidance head however inpainting intrinsic multi-head optical optical flow processing query transformer transformers type video video processing