THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation (2024-02-13T00:00:00.000000Z)

TL;DR

The COLOSSEUM is presented, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations, and identifies that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most.

Abstract

To realize effective large-scale, real-world robotic applications, we must evaluate how well our robot policies adapt to changes in environmental conditions. Unfortunately, a majority of studies evaluate robot performance in environments closely resembling or even identical to the training setup. We present THE COLOSSEUM, a novel simulation benchmark, with 20 diverse manipulation tasks, that enables systematical evaluation of models across 14 axes of environmental perturbations. These perturbations include changes in color, texture, and size of objects, table-tops, and backgrounds; we also vary lighting, distractors, physical properties perturbations and camera pose. Using THE COLOSSEUM, we compare 5 state-of-the-art manipulation models to reveal that their success rate degrades between 30-50% across these perturbation factors. When multiple perturbations are applied in unison, the success rate degrades $\geq$75%. We identify that changing the number of distractor objects, target object color, or lighting conditions are the perturbations that reduce model performance the most. To verify the ecological validity of our results, we show that our results in simulation are correlated ($\bar{R}^2 = 0.614$) to similar perturbations in real-world experiments. We open source code for others to use THE COLOSSEUM, and also release code to 3D print the objects used to replicate the real-world perturbations. Ultimately, we hope that THE COLOSSEUM will serve as a benchmark to identify modeling decisions that systematically improve generalization for manipulation. See https://robot-colosseum.github.io/ for more details.

Authors

Jiafei Duan

3 papers

Ranjay Krishna

3 papers

Wilbert Pumacay

1 papers

TL;DR

Abstract

Authors

References78 items

Octo: An Open-Source Generalist Robot Policy

FMB: A functional manipulation benchmark for generalizable robotic learning

PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play

MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations

Learning Generalizable Manipulation Policies with Object-Centric 3D Representations

Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots

MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning

Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0

GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

RL-ViGen: A Reinforcement Learning Benchmark for Visual Generalization

VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models

Decomposing the Generalization Gap in Imitation Learning for Visual Robotic Manipulation

Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation

KITE: Keypoint-Conditioned Policies for Semantic Manipulation

RVT: Robotic View Transformer for 3D Object Manipulation

AR2-D2: Training a Robot Without a Robot

LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning

Imitating Task and Motion Planning with Visuomotor Transformers

FurnitureBench: Reproducible real-world benchmark for long-horizon complex manipulation

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware

Visual Instruction Tuning

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning

Diffusion policy: Visuomotor policy learning via action diffusion

MimicPlay: Long-Horizon Imitation Learning by Watching Human Play

GenAug: Retargeting behaviors to unseen situations via Generative Augmentation

RT-1: Robotics Transformer for Real-World Control at Scale

Is Conditional Generative Modeling all you need for Decision-Making?

A taxonomy and review of generalization research in NLP

Real-World Robot Learning with Masked Visual Pre-training

Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation

VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation

ProcTHOR: Large-Scale Embodied AI Using Procedural Generation

Reinforcement Learning with Neural Radiance Fields

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

R3M: A Universal Visual Representation for Robot Manipulation

Ego4D: Around the World in 3,000 Hours of Egocentric Video

Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets

CLIPort: What and Where Pathways for Robotic Manipulation

Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning

Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning

Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisation

Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulation

Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes

Learning Transferable Visual Models From Natural Language Supervision

Generalization in Reinforcement Learning by Soft Data Augmentation

RetinaGAN: An Object-aware Approach to Sim-to-Real Transfer

Learning Continuous Control Actions for Robotic Grasping with Reinforcement Learning

robosuite: A Modular Simulation Framework and Benchmark for Robot Learning

Reinforcement Learning with Augmented Data

RoboNet: Large-Scale Multi-Robot Learning

RLBench: The Robot Learning Benchmark & Learning Environment

PyRep: Bringing V-REP to Deep Robot Learning

arXiv

Active Domain Randomization

Transferring Visuomotor Learning from Simulation to the Real World for Robotics Manipulation Tasks

QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation

Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation

Proximal Policy Optimization Algorithms

Domain randomization for transferring deep neural networks from simulation to the real world

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations

The YCB object and Model set: Towards common benchmarks for manipulation research

Microsoft COCO: Common Objects in Context

Playing Atari with Deep Reinforcement Learning

MuJoCo: A physics engine for model-based control

A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning

Multilayer perceptron and neural networks

ImageNet: A large-scale hierarchical image database

Learning representations by back-propagating errors

Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World

Qwen-VL: A Frontier Large Vision-Language Model with Versatile Abilities

ActioNet: An Interactive End-To-End Platform For Task-Based Data Collection And Augmentation In 3D Environment

ALVINN, an autonomous land vehicle in a neural network

V-REP: A versatile and scalable robot simulation framework

Lauderdale, FL, USA, 11–13 Apr

Autonomous Inverted Helicopter Flight via Reinforcement Learning