Fine-tuning and Sampling Strategies for Multimodal Role Labeling of Entities under Class Imbalance

in Scientific publications
Share this publication
Author Syrielle Montariol, Étienne Simon, Arij Riabi, Djamé Seddah
Title of Journal, Proc. or Book Proceedings of the Workshop on Combating Online Hostile Posts in Regional Languages during Emergency Situations
DOI 10.18653/v1/2022.constraint-1.7
Repository link
Peer reviewed Yes
Open access Yes


We propose our solution to the multimodal semantic role labeling task from the CONSTRAINT’22 workshop. The task aims at classifying entities in memes into classes such as “hero” and “villain”. We use several pre-trained multi-modal models to jointly encode the text and image of the memes, and mplement three systems to classify the role of the entities. We propose dynamic sampling strategies to tackle the issue of class imbalance. Finally, we perform qualitative analysis on the representations of the entities.

Previous Post
TRA-I and radicalisation processes: current considerations and future perspectives
Next Post
Tâches Auxiliaires Multilingues pour le Transfert de Modèles de Détection de Discours Haineux (Multilingual Auxiliary Tasks for Zero-Shot Cross-Lingual Transfer of Hate Speech Detection)
You may also be interested in these topics
Skip to content