We use a few cookies

Essential cookies keep the library working. With your permission, we'd also like to use analytics to see what's useful and improve the experience. Read our Privacy Policy or Cookie Policy.

Skip to main content
Home/e-Resources/Artificial Intelligence & Machine Learning/Multimodal Language Analysis with Recurrent Multistage Fusion
Conference ProceedingsArtificial Intelligence & Machine Learning

Multimodal Language Analysis with Recurrent Multistage Fusion

Checking your access to this record.

Abstract & Overview

Computational modeling of human multimodal language is an emerging research area in natural language processing spanning the language, visual and acoustic modalities.Comprehending multimodal language requires modeling not only the interactions within each modality (intra-modal interactions) but more importantly the interactions between modalities (cross-modal interactions).In this paper, we propose the Recurrent Multistage Fusion Network (RMFN) which decomposes the fusion problem into multiple stages, each of them focused on a subset of multimodal signals for specialized, effective fusion.Crossmodal interactions are modeled using this multistage fusion approach which builds upon intermediate representations of previous stages.Temporal and intra-modal interactions are modeled by integrating our proposed fusion approach with a system of recurrent neural networks.The RMFN displays state-of-the-art performance in modeling human multimodal language across three public datasets relating to multimodal sentiment analysis, emotion recognition, and speaker traits recognition.We provide visualizations to show that each stage of fusion focuses on a different subset of multimodal signals, learning increasingly discriminative multimodal representations.

Cite this record

Paul Pu Liang, Liu Ziyin, AmirAli Bagher Zadeh, Louis-Philippe Morency (2018). Multimodal Language Analysis with Recurrent Multistage Fusion. Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653/v1/d18-1014

Copy in another style

Liang, P. P., Ziyin, L., Zadeh, A. B., & Morency, L. (2018). Multimodal Language Analysis with Recurrent Multistage Fusion. Conference on Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653/v1/d18-1014
Author names are used as deposited, so check their order before you submit.

Metadata is harvested from open-access indexes and can be incomplete. Check a citation against the published version before relying on it — see our open access & attribution policy.