Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space (2022-06-23T00:00:00.000000Z)

TL;DR

A 3D Token Representation Layer (3DTRL) is proposed that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations and outperform their backbone Transformers in all the tasks with minimal added computation.

Abstract

Humans are remarkably flexible in understanding viewpoint changes due to visual cortex supporting the perception of 3D structure. In contrast, most of the computer vision models that learn visual representation from a pool of 2D images often fail to generalize over novel camera viewpoints. Recently, the vision architectures have shifted towards convolution-free architectures, visual Transformers, which operate on tokens derived from image patches. However, these Transformers do not perform explicit operations to learn viewpoint-agnostic representation for visual understanding. To this end, we propose a 3D Token Representation Layer (3DTRL) that estimates the 3D positional information of the visual tokens and leverages it for learning viewpoint-agnostic representations. The key elements of 3DTRL include a pseudo-depth estimator and a learned camera matrix to impose geometric transformations on the tokens, trained in an unsupervised fashion. These enable 3DTRL to recover the 3D positional information of the tokens from 2D patches. In practice, 3DTRL is easily plugged-in into a Transformer. Our experiments demonstrate the effectiveness of 3DTRL in many vision tasks including image classification, multi-view video alignment, and action recognition. The models with 3DTRL outperform their backbone Transformers in all the tasks with minimal added computation. Our code is available at https://github.com/elicassion/3DTRL.

Authors

M. Ryoo

11 papers

Srijan Das

2 papers

Jinghuan Shang

1 papers

TL;DR

Abstract

Authors

References69 items

StARformer: Transformer With State-Action-Reward Representations for Robot Learning

Neural Neural Textures Make Sim2Real Consistent

Learning Multi-View Aggregation In the Wild for Large-Scale 3D Semantic Segmentation

Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Vision-Based Manipulators Need to Also See from Their Hands

Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic Manipulation

A ConvNet for the 2020s

ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints

Self-supervised Video Transformer

MonoScene: Monocular 3D Semantic Scene Completion

SWAT: Spatial Structure Within and Among Tokens

4D-Net for Learned Multi-Modal Alignment

What Matters in Learning from Offline Human Demonstrations for Robot Manipulation

Self-Supervised Disentangled Representation Learning for Third-Person Imitation Learning

Eliminating topological errors in neural network rotation estimation using self-selecting ensembles

Video Swin Transformer

TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?

CoAtNet: Marrying Convolution and Attention for All Data Sizes

ImVoxelNet: Image to Voxels Projection for Monocular and Multi-View General-Purpose 3D Object Detection

VPN++: Rethinking Video-Pose Embeddings for Understanding Activities of Daily Living

Representation Learning via Global Temporal Alignment and Cycle-Consistency

LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference

Recognizing Actions in Videos from Unseen Viewpoints

ViViT: A Video Vision Transformer

CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification

Planar Surface Reconstruction from Sparse Views

Transformer in Transformer

Is Space-Time Attention All You Need for Video Understanding?

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Training data-efficient image transformers & distillation through attention

Point Transformer

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

VPN: Learning Video-Pose Embedding for Activities of Daily Living

View-Invariant Probabilistic Embedding for Human Pose

SuperGlue: Learning Feature Matching With Graph Neural Networks

Toyota Smarthome: Real-World Activities of Daily Living

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features

Temporal Cycle-Consistency Learning

Learning Correspondence From the Cycle-Consistency of Time

On the Continuity of Rotation Representations in Neural Networks

Learning Actionable Representations from Visual Observations

Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos

Actor and Observer: Joint Modeling of First and Third-Person Videos

mixup: Beyond Empirical Risk Minimization

Attention is All you Need

The Kinetics Human Action Video Dataset

SfM-Net: Learning of Structure and Motion from Video

Time-Contrastive Networks: Self-Supervised Learning from Video

Third-Person Imitation Learning

Perspective Transformer Nets: Learning Single-View 3D Object Reconstruction without 3D Supervision

NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis

Shuffle and Learn: Unsupervised Learning Using Temporal Order Verification

Towards Viewpoint Invariant 3D Human Pose Estimation

Deep Residual Learning for Image Recognition

Rethinking the Inception Architecture for Computer Vision

Spatial Transformer Networks

Return of the Devil in the Details: Delving Deep into Convolutional Nets

ImageNet: A large-scale hierarchical image database

Convolutional networks for images, speech, and time series

Generalized procrustes analysis

A NEW MEASURE OF RANK CORRELATION

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

StARformer: Transformer with State-Action-Reward Representations

Mesh R-CNN

ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models

Learning Multiple Layers of Features from Tiny Images

Pybullet, a python module for physics simulation for games, robotics and machine learning

Representing Scenes as Neural Radiance Fields for View Synthesis

Field of Study

Journal Information

Name

Volume

Venue Information

Name

Type