Technical Report for ICCV 2021 Challenge SSLAD-Track3B: Transformers Are Better Continual Learners. (arXiv:2201.04924v1 [cs.CV])

In the SSLAD-Track 3B challenge on continual learning, we propose the method
of COntinual Learning with Transformer (COLT). We find that transformers suffer
less from catastrophic forgetting compared to convolutional neural network. The
major principle of our method is to equip the transformer based feature
extractor with old knowledge distillation and head expanding strategies to
compete catastrophic forgetting. In this report, we first introduce the overall
framework of continual learning for object detection. Then, we analyse the key
elements' effect …

arxiv cv iccv report transformers