Visual Goal-Step Inference using wikiHow (2021-04-12T00:00:00.000000Z)

TL;DR

This work proposes the Visual Goal-Step Inference task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal, and shows that the task is challenging for state-of-the-art multimodal models.

Abstract

Understanding what sequence of steps are needed to complete a goal can help artificial intelligence systems reason about human activities. Past work in NLP has examined the task of goal-step inference for text. We introduce the visual analogue. We propose the Visual Goal-Step Inference (VGSI) task, where a model is given a textual goal and must choose which of four images represents a plausible step towards that goal. With a new dataset harvested from wikiHow consisting of 772,277 images representing human actions, we show that our task is challenging for state-of-the-art multimodal models. Moreover, the multimodal representation learned from our data can be effectively transferred to other datasets like HowTo100m, increasing the VGSI accuracy by 15 - 20%. Our task will facilitate multimodal reasoning about procedural events.

Authors

Mark Yatskar

12 papers

Chris Callison-Burch

13 papers

Yue Yang

1 papers

TL;DR

Abstract

Authors

References30 items

Connecting the Dots: Event Graph Schema Induction with Path Language Modeling

Reasoning about Goals, Steps, and Temporal Ordering with WikiHow

Intent Detection with WikiHow

Recent Trends in Natural Language Understanding for Procedural Knowledge

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Learning Household Task Knowledge from WikiHow Descriptions

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips

COIN: A Large-Scale Dataset for Comprehensive Instructional Video Analysis

Joint Slot Filling and Intent Detection via Capsule Neural Networks

SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference

Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces

Learning Procedures from Text: Codifying How-to Procedures in Deep Neural Networks

Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering

Learning Two-Branch Neural Networks for Image-Text Matching Tasks

Billion-Scale Similarity Search with GPUs

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation

Show and Tell: Lessons Learned from the 2015 MSCOCO Image Captioning Challenge

Gaussian Error Linear Units (GELUs)

Rethinking the Inception Architecture for Computer Vision

DenseCap: Fully Convolutional Localization Networks for Dense Captioning

Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models

Adam: A Method for Stochastic Optimization

Deep Metric Learning Using Triplet Network

Deep Fragment Embeddings for Bidirectional Image Sentence Mapping

DeViSE: A Deep Visual-Semantic Embedding Model

Reporting bias and knowledge acquisition

ImageNet: A large-scale hierarchical image database

Future is not One-dimensional: Graph Modeling based Complex Event Schema Induction for Event Prediction

2019) is a multimodal encoder that aims to ground text to images. It takes as an input an image I and a related sentence G

Field of Study

Venue Information

Name

Type

URL

Alternate Names