One-Shot Object Affordance Detection in the Wild (2021-08-08T00:00:00.000000Z)

TL;DR

A One-Shot Affordance Detection Network (OSAD-Net) is devised that firstly estimates the human action purpose and then transfers it to help detect the common affordance from all candidate images and builds a large-scale purpose-driven affordance dataset v2.

Abstract

Affordance detection refers to identifying the potential action possibilities of objects in an image, which is a crucial ability for robot perception and manipulation. To empower robots with this ability in unseen scenarios, we first study the challenging one-shot affordance detection problem in this paper, i.e., given a support image that depicts the action purpose, all objects in a scene with the common affordance should be detected. To this end, we devise a One-Shot Affordance Detection Network (OSAD-Net) that firstly estimates the human action purpose and then transfers it to help detect the common affordance from all candidate images. Through collaboration learning, OSAD-Net can capture the common characteristics between objects having the same underlying affordance and learn a good adaptation capability for perceiving unseen affordances. Besides, we build a large-scale purpose-driven affordance dataset v2 (PADv2) by collecting and labeling 30k images from 39 affordance and 103 object categories. With complex scenes and rich annotations, our PADv2 dataset can be used as a test bed to benchmark affordance detection methods and may also facilitate downstream vision tasks, such as scene understanding, action recognition, and robot manipulation. Specifically, we conducted comprehensive experiments on PADv2 dataset by including 11 advanced models from several related research fields. Experimental results demonstrate the superiority of our model over previous representative ones in terms of both objective metrics and visual quality. The benchmark suite is available at https://github.com/lhc1224/OSAD_Net.

Authors

D. Tao

37 papers

Wei Zhai

4 papers

Hongcheng Luo

3 papers

TL;DR

Abstract

Authors

References119 items

Learning Affordance Grounding from Exocentric Images

Phrase-Based Affordance Detection via Cyclic Bilateral Interaction

ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond

Background Activation Suppression for Weakly Supervised Object Localization

Learning Visual Affordance Grounding From Demonstration Videos

One-Shot Affordance Detection

ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias

MinMaxCAM: Improving object coverage for CAM-basedWeakly Supervised Object Localization

Adaptive Prototype Learning and Allocation for Few-Shot Segmentation

3D AffordanceNet: A Benchmark for Visual Object Affordance Understanding

Deep Gaussian Processes for Few-Shot Segmentation

TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localization

Unveiling the Potential of Structure Preserving for Weakly Supervised Object Localization

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

Variational Prototype Inference for Few-Shot Semantic Segmentation

Progressive One-shot Human Parsing

Empowering Things With Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Recursive Context Routing for Object Detection

Polysemy Deciphering Network for Robust Human–Object Interaction Detection

Prior Guided Feature Enrichment Network for Few-Shot Segmentation

View Transfer on Human Skeleton Pose: Automatically Disentangle the View-Variant and View-Invariant Information for Pose Representation Learning

Re-Thinking Co-Salient Object Detection

Intention-Related Natural Language Grounding via Object Affordance Detection and Intention Semantic Extraction

Self-Supervised Tuning for Few-Shot Segmentation

Adversarial Feature Hallucination Networks for Few-Shot Learning

Highly Efficient Salient Object Detection with 100K Parameters

Towards High Performance Human Keypoint Detection

An Exploration of Embodied Visual Exploration

Residual Objectness for Imbalance Reduction

EGNet: Edge Guidance Network for Salient Object Detection

Deep High-Resolution Representation Learning for Visual Recognition

IoU-balanced Loss Functions for Single-stage Object Detection

Expectation-Maximization Attention Networks for Semantic Segmentation

DR Loss: Improving Object Detection by Distributional Ranking

Object affordance detection with relationship-aware network

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer

BASNet: Boundary-Aware Salient Object Detection

One-Shot Texture Retrieval with Global Context Metric

Cascaded Partial Decoder for Fast and Accurate Salient Object Detection

Putting Humans in a Scene: Learning Affordance in 3D Indoor Environments

CANet: Class-Agnostic Segmentation Networks With Iterative Refinement and Attentive Few-Shot Learning

Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation

Grounded Human-Object Interaction Hotspots From Video

Interact as You Intend: Intention-Driven Human-Object Interaction Detection

Meta-Learning with Latent Embedding Optimization

Learning task-oriented grasping for tool manipulation from simulated self-supervision

Demo2Vec: Reasoning Object Affordances from Online Videos

Enhanced-alignment Measure for Binary Foreground Map Evaluation

Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation

Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition

Learning to Act Properly: Predicting and Explaining Affordances from Images

Learning to Compare: Relation Network for Few-Shot Learning

A brief review of affordance in robotic manipulation research

AffordanceNet: An End-to-End Deep Learning Approach for Object Affordance Detection

Object-based affordances detection with Convolutional Neural Networks and dense Conditional Random Fields

One-Shot Learning for Semantic Segmentation

Predicting Human Activities Using Stochastic Grammar

Inferring Human Attention by Learning Latent Intentions

Weakly Supervised Affordance Detection

Scene Parsing through ADE20K Dataset

Adaptive Binarization for Weakly Supervised Affordance Segmentation

Binge Watching: Scaling Affordance Learning from Sitcoms

Rethinking Atrous Convolution for Semantic Image Segmentation

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

Deep Affordance-Grounded Sensorimotor Object Recognition

Prototypical Networks for Few-shot Learning

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks

Pyramid Scene Parsing Network

Optimization as a Model for Few-Shot Learning

Learning to Learn: Model Regression Networks for Easy Small Sample Learning

Semi-Supervised Classification with Graph Convolutional Networks

Matching Networks for One Shot Learning

Generative Adversarial Imitation Learning

A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation

Learning to Detect Visual Grasp Affordance