Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions (2022-09-07T00:00:00.000000Z)

TL;DR

An overview of the computational and theoretical foundations of multimodal machine learning is provided, defining three key principles of modality heterogeneity, connections, and interactions that have driven subsequent innovations, and proposing a taxonomy of six core technical challenges covering historical and recent trends.

Abstract

Multimodal machine learning is a vibrant multi-disciplinary research field that aims to design computer agents with intelligent capabilities such as understanding, reasoning, and learning through integrating multiple communicative modalities, including linguistic, acoustic, visual, tactile, and physiological messages. With the recent interest in video understanding, embodied autonomous agents, text-to-image generation, and multisensor fusion in application domains such as healthcare and robotics, multimodal machine learning has brought unique computational and theoretical challenges to the machine learning community given the heterogeneity of data sources and the interconnections often found between modalities. However, the breadth of progress in multimodal research has made it difficult to identify the common themes and open questions in the field. By synthesizing a broad range of application domains and theoretical frameworks from both historical and recent perspectives, this article is designed to provide an overview of the computational and theoretical foundations of multimodal machine learning. We start by defining three key principles of modality heterogeneity, connections, and interactions that have driven subsequent innovations, and propose a taxonomy of six core technical challenges: representation, alignment, reasoning, generation, transference, and quantification covering historical and recent trends. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches. We end by motivating several open problems for future research as identified by our taxonomy.

Authors

Paul Pu Liang

8 papers

Amir Zadeh

5 papers

Louis-philippe Morency

12 papers

TL;DR

Abstract

Authors

References435 items

MultiIoT: Benchmarking Machine Learning for the Internet of Things

Lecture Presentations Multimodal Dataset: Towards Understanding Multimodality in Educational Videos

An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

WebArena: A Realistic Web Environment for Building Autonomous Agents

Multimodal Distillation for Egocentric Action Recognition

Kosmos-2: Grounding Multimodal Large Language Models to the World

Factorized Contrastive Learning: Going Beyond Multi-view Redundancy

Multimodal Web Navigation with Instruction-Finetuned Foundation Models

InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning

LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model

PaLM-E: An Embodied Multimodal Language Model

Multimodal Prompting with Missing Modalities for Visual Recognition

Quantifying & Modeling Multimodal Interactions: An Information Decomposition Framework

MusicLM: Generating Music From Text

Make-A-Video: Text-to-Video Generation without Text-Video Data

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Do Androids Laugh at Electric Sheep? Humor “Understanding” Benchmarks from The New Yorker Caption Contest

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

Multimodal biomedical AI

Face-to-Face Contrastive Learning for Social Intelligence Question-Answering

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

MultiViz: Towards Visualizing and Understanding Multimodal Models

Multimodal Learning With Transformers: A Survey

The Modality Focusing Hypothesis: Towards Understanding Crossmodal Knowledge Distillation

Deep Learning in Multimodal Remote Sensing Data Fusion: A Comprehensive Review

Flamingo: a Visual Language Model for Few-Shot Learning

Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion

Brainish: Formalizing A Multimodal Language for Intelligence and Consciousness

Are Multimodal Transformers Robust to Missing Modality?

Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

VL-InterpreT: An Interactive Visualization Tool for Interpreting Vision-Language Transformers

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning

High-Modality Multimodal Transformer: Quantifying Modality & Interaction Heterogeneity for High-Modality Representation Learning

Multi-Modal Knowledge Graph Construction and Application: A Survey

Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks

Pre-Trained Language Models for Interactive Decision-Making

Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents

A Survey of Controllable Text Generation Using Transformer-based Pre-trained Language Models

CLIP-Event: Connecting Text and Images with Event Structures

High-Resolution Image Synthesis with Latent Diffusion Models

KAT: A Knowledge Augmented Transformer for Vision-and-Language

Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text

FLAVA: A Foundational Language And Vision Alignment Model

PolyViT: Co-training Vision Transformers on Images, Videos and Audio

Perceptual Score: What Data Modalities Does Your Model Perceive?

Multimodal Learning using Optimal Transport for Sarcasm and Humor Detection

Towards Reliable Multimodal Stress Detection under Distribution Shift

Multimodal datasets: misogyny, pornography, and malignant stereotypes

Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generation

Multimodal Behavior Modeling for Socially Interactive Agents

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning

A theory of consciousness from a theoretical computer science perspective: Insights from the Conscious Turing Machine

M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

The Bottleneck

Multimodal Few-Shot Learning with Frozen Language Models

Learning Language and Multimodal Privacy-Preserving Markers of Mood from Mobile Data

Cross-Modal Discrete Representation Learning

What Makes Multimodal Learning Better than Single (Provably)

Grounding ‘Grounding’ in NLP

MERLOT: Multimodal Neural Script Knowledge Models

GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Humor Knowledge Enriched Transformer for Understanding Multimodal Humor

Multi-View Representation Learning with Manifold Smoothness

Leveraging Sparse Linear Layers for Debuggable Deep Networks

MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences

Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges

Multimodal Clustering Networks for Self-supervised Learning from Unlabeled Videos

VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text

Worst of Both Worlds: Biases Compound in Pre-trained Vision-and-Language Models

The Power of Scale for Parameter-Efficient Prompt Tuning