Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks

Effectively leveraging multimodal information from social media posts is essential to various downstream tasks such as sentiment analysis, sarcasm detection or hate speech classification. Jointly modeling text and images is challenging because cross-modal semantics might be hidden or the relation between image and text is weak. However, prior work on multimodal classification of social media posts has not yet addressed these challenges. In this work, we present an extensive study on the effectiveness of using two auxiliary losses jointly with …

analysis classification cs.cl cs.lg cs.si detection hate speech hidden image images information media modal modeling multimodal prior semantics sentiment sentiment analysis social social media speech tasks text