Text or Image? What is More Important in Cross-Domain Generalization Capabilities of Hate Meme Detection Models?

This paper delves into the formidable challenge of cross-domain generalization in multimodal hate meme detection, presenting compelling findings. We provide enough pieces of evidence supporting the hypothesis that only the textual component of hateful memes enables the existing multimodal classifier to generalize across different domains, while the image component proves highly sensitive to a specific training dataset. The evidence includes demonstrations showing that hate-text classifiers perform similarly to hate-meme classifiers in a zero-shot setting. Simultaneously, the introduction of captions generated …

capabilities challenge classifier cs.ai cs.cl cs.cv detection domain evidence hypothesis image meme memes multimodal paper presenting text textual