Collaborative Transformers for Grounded Situation Recognition (2022-03-30T00:00:00.000000Z)

TL;DR

A novel approach where the two processes for activity classification and entity estimation are interactive and complementary, which achieves the state of the art in all evaluation metrics on the SWiG dataset.

Abstract

Grounded situation recognition is the task of predicting the main activity, entities playing certain roles within the activity, and bounding-box groundings of the entities in the given image. To effectively deal with this challenging task, we introduce a novel approach where the two processes for activity classification and entity estimation are interactive and complementary. To implement this idea, we propose Collaborative Glance-Gaze TransFormer (CoFormer) that consists of two modules: Glance transformer for activity classification and Gaze transformer for entity estimation. Glance transformer predicts the main activity with the help of Gaze transformer that analyzes entities and their relations, while Gaze transformer estimates the grounded entities by focusing only on the entities relevant to the activity predicted by Glance transformer. Our CoFormer achieves the state of the art in all evaluation metrics on the SWiG dataset. Training code and model weights are available at https://github.com/jhcho99/CoFormer.

Authors

Junhyeong Cho

3 papers

Youngseok Yoon

2 papers

Suha Kwak

5 papers

TL;DR

Abstract

Authors

References44 items

Grounded Situation Recognition with Transformers

Context-aware Scene Graph Generation with Seq2Seq Transformers

CTRL-C: Camera calibration TRansformer with Line-Classification

Paint Transformer: Feed Forward Neural Painting with Stroke Prediction

Segmentation-grounded Scene Graph Generation

HOTR: End-to-End Human-Object Interaction Detection with Transformers

Human-like Controllable Image Captioning with Verb-specific Semantic Roles

Spatially Conditioned Graphs for Detecting Human–Object Interactions

Attention-Based Context Aware Reasoning for Situation Recognition

End-to-End Object Detection with Transformers

Learning Human-Object Interaction Detection Using Interaction Points

Grounded Situation Recognition

Normalized and Geometry-Aware Self-Attention Network for Image Captioning

On Layer Normalization in the Transformer Architecture

Mixture-Kernel Graph Attention Network for Situation Recognition

Attention on Attention for Image Captioning

Habit Formation and Behavior Change

Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression

Transferable Interactiveness Knowledge for Human-Object Interaction Detection

Graph R-CNN for Scene Graph Generation

Decoupled Weight Decay Regularization

Situation Recognition with Graph Neural Networks

Focal Loss for Dense Object Detection

Focal Loss for Dense Object Detection

Attention is All you Need

Recurrent Models for Situation Recognition

Scene Graph Generation by Iterative Message Passing

Feature Pyramid Networks for Object Detection

Commonly Uncommon: Semantic Sparsity in Situation Recognition

Situation Recognition: Visual Semantic Role Labeling for Image Understanding

Image Captioning with Semantic Attention

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Gated Graph Sequence Neural Networks

Show and tell: A neural image caption generator

Very Deep Convolutional Networks for Large-Scale Image Recognition

Understanding the difficulty of training deep feedforward neural networks

ImageNet: A large-scale hierarchical image database

Maps of Bounded Rationality: Psychology for Behavioral Economics

Background to Framenet

Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data

Long Short-Term Memory

Oxford research encyclopedia of psychology

PSYCHOLOGICAL SCIENCE Research Article Numeracy and Decision Making

Field of Study

Journal Information

Name

Page

Venue Information

Name

Type

URL

Alternate Names