Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable

arXiv:2405.00466v1 Announce Type: new
Abstract: Foundational generative models should be traceable to protect their owners and facilitate safety regulation. To achieve this, traditional approaches embed identifiers based on supervisory trigger-response signals, which are commonly known as backdoor watermarks. They are prone to failure when the model is fine-tuned with nontrigger data. Our experiments show that this vulnerability is due to energetic changes in only a few 'busy' layers during fine-tuning. This yields a novel arbitrary-in-arbitrary-out (AIAO) strategy that makes watermarks …

abstract arxiv backdoor cs.cr cs.cv data diffusion diffusion models embed failure foundational generative generative models lazy protect regulation safety traceable type watermarks