Multi-Person 3D Motion Prediction with Multi-Range Transformers (2021-11-23T00:00:00.000000Z)

TL;DR

A Multi-Range Transformers model is introduced which contains of a local-range encoder for individual motion and a global-rangeEncoder for social interactions, which outperforms state-of-the-art methods on long-term 3D motion prediction, but also generates diverse social interactions.

Abstract

We propose a novel framework for multi-person 3D motion trajectory prediction. Our key observation is that a human's action and behaviors may highly depend on the other persons around. Thus, instead of predicting each human pose trajectory in isolation, we introduce a Multi-Range Transformers model which contains of a local-range encoder for individual motion and a global-range encoder for social interactions. The Transformer decoder then performs prediction for each person by taking a corresponding pose as a query which attends to both local and global-range encoder features. Our model not only outperforms state-of-the-art methods on long-term 3D motion prediction, but also generates diverse social interactions. More interestingly, our model can even predict 15-person motion simultaneously by automatically dividing the persons into different interaction groups. Project page with code is available at https://jiashunwang.github.io/MRT/.

Authors

Xiaolong Wang

9 papers

Huazhe Xu

6 papers

Jiashun Wang

3 papers

TL;DR

Abstract

Authors

References76 items

Pose Recognition with Cascade Transformers

Action-Conditioned 3D Human Motion Synthesis with Transformer VAE

TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild

SimPoE: Simulated Character Control for 3D Human Pose Estimation

TFPose: Direct Human Pose Estimation with Transformers

End-to-End Trainable Multi-Instance Pose Estimation with Transformers

Three Steps to Multimodal Trajectory Prediction: Modality Clustering, Classification and Synthesis

Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenes

We are More than Our Joints: Predicting how 3D Bodies Move

Learning to Generate Diverse Dance Motions with Transformer

History Repeats Itself: Human Motion Prediction via Motion Attention

Socially and Contextually Aware Human Motion and Pose Forecasting

Generative Pretraining From Pixels

Long-term Human Motion Prediction with Scene Context

Language Models are Few-Shot Learners

End-to-End Object Detection with Transformers

Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction

Recursive Social Behavior Graph for Trajectory Prediction

Attention, please: A Spatio-temporal Transformer for 3D Human Motion Prediction

It Is Not the Journey but the Destination: Endpoint Conditioned Trajectory Prediction

Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction

VIBE: Video Inference for Human Body Pose and Shape Estimation

Generating 3D People in Scenes Without People

Learning Trajectory Dependencies for Human Motion Prediction

Predicting 3D Human Dynamics From Video

Diverse Trajectory Forecasting with Determinantal Point Processes

Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks

Generating Long Sequences with Sparse Transformers

Social Ways: Learning Multi-Modal Distributions of Pedestrian Trajectories With GANs

VideoBERT: A Joint Model for Video and Language Representation Learning

Human Motion Prediction via Spatio-Temporal Inpainting

A Neural Temporal Model for Human Motion Prediction

Adversarial Geometry-Aware Human Motion Prediction

SoPhie: An Attentive GAN for Predicting Paths Compliant to Social and Physical Constraints

Convolutional Sequence to Sequence Model for Human Dynamics

QuaterNet: A Quaternion-based Recurrent Model for Human Motion

Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks

Image Transformer

Single-Shot Multi-person 3D Pose Estimation from Monocular RGB

Relation Networks for Object Detection

Non-local Neural Networks

Auto-Conditioned Recurrent Networks for Extended Complex Human Motion Synthesis

Attention is All you Need

On Human Motion Prediction Using Recurrent Neural Networks

DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents

Learning Human Motion Models for Long-Term Predictions

Learning social affordance grammar from videos: Transferring human interactions to human-robot interactions

Deep Representation Learning for Human Motion Prediction and Classification

Soft + Hardwired Attention: An LSTM Framework for Human Trajectory Prediction and Abnormal Event Detection

Social LSTM: Human Trajectory Prediction in Crowded Spaces

Learning Social Affordance for Human-Robot Interaction

Panoptic Studio: A Massively Multiview System for Social Motion Capture

Structural-RNN: Deep Learning on Spatio-Temporal Graphs

Learning motion manifolds with convolutional autoencoders

Recurrent Network Models for Human Dynamics

Adam: A Method for Stochastic Optimization

Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling

On the Properties of Neural Machine Translation: Encoder–Decoder Approaches

Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments

Socially-Aware Large-Scale Crowd Forecasting

Understanding collective crowd behaviors: Learning a Mixture model of Dynamic pedestrian-Agents

On discrete cosine transform

Who are you with and where are you going?

You'll never walk alone: Modeling social behavior for multi-target tracking

Abnormal crowd behavior detection using social force model

Long Short-Term Memory

Social force model for pedestrian dynamics.

Does the chimpanzee have a theory of mind?

Learn to Dance with AIST++: Music Conditioned 3D Dance Generation

TransPose: Towards Explainable Human Pose Estimation by Transformer

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Language Models are Unsupervised Multitask Learners

Set transformer: A framework for attention-based permutation-invariant neural networks

Supplementary Material to: Recovering Accurate 3D Human Pose in The Wild Using IMUs and a Moving Camera

Cmu graphics lab motion capture database

Ross , and Angjoo Kanazawa