On the Effectiveness of Distillation in Mitigating Backdoors in Pre-trained Encoder

arXiv:2403.03846v1 Announce Type: new
Abstract: In this paper, we study a defense against poisoned encoders in SSL called distillation, which is a defense used in supervised learning originally. Distillation aims to distill knowledge from a given model (a.k.a the teacher net) and transfer it to another (a.k.a the student net). Now, we use it to distill benign knowledge from poisoned pre-trained encoders and transfer it to a new encoder, resulting in a clean pre-trained encoder. In particular, we conduct an …

abstract arxiv cs.lg defense distillation encoder knowledge paper ssl study supervised learning transfer type