Multimodal Material Classification for Robots using Spectroscopy and High Resolution Texture Imaging (2020-04-02T00:00:00.000000Z)

TL;DR

A multimodal sensing technique, leveraging near-infrared spectroscopy and close-range high resolution texture imaging, that enables robots to estimate the materials of household objects and a neural network architecture that learns a compact multi-modal representation of spectral measurements and texture images is presented.

Abstract

Material recognition can help inform robots about how to properly interact with and manipulate real-world objects. In this paper, we present a multimodal sensing technique, leveraging near-infrared spectroscopy and close-range high resolution texture imaging, that enables robots to estimate the materials of household objects. We release a dataset of high resolution texture images and spectral measurements collected from a mobile manipulator that interacted with 144 house-hold objects. We then present a neural network architecture that learns a compact multimodal representation of spectral measurements and texture images. When generalizing material classification to new objects, we show that this multimodal representation enables a robot to recognize materials with greater performance as compared to prior state-of-the-art approaches. Finally, we present how a robot can combine this high resolution local sensing with images from the robot’s head-mounted camera to achieve accurate material classification over a scene of objects on a table.

Authors

C. Kemp

6 papers

Zackory Erickson

4 papers

S. Chernova

5 papers

TL;DR

Abstract

Authors

References75 items

Tool Substitution with Shape and Material Reasoning Using Dual Neural Networks

Leveraging Multimodal Haptic Sensory Data for Robust Cutting

CAGE: Context-Aware Grasping Engine

Autonomous Tool Construction Using Part Shape and Attachment Prediction

EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

Toward Fingertip Non-Contact Material Recognition and Near-Distance Ranging for Robotic Grasping

Feasibility study of smartphone-based Near Infrared Spectroscopy (NIRS) for salted minced meat composition diagnostics at different temperatures.

Automated Recycling Separation Enabled by Soft Robotic Material Classification

“Touching to See” and “Seeing to Feel”: Robotic Cross-modal Sensory Data Generation for Visual-Tactile Perception

Tactile Sensors for Friction Estimation and Incipient Slip Detection—Toward Dexterous Robotic Manipulation: A Review

Portable Spectroscopy

Near infrared spectroscopy: A mature analytical technique with new perspectives - A review.

Material Recognition Using a Capacitive Proximity Sensor with Flexible Spatial Resolution

Quantitative analysis of a pharmaceutical formulation: Performance comparison of different handheld near‐infrared spectrometers

ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness

Learn to Combine Modalities in Multimodal Deep Learning

Voronoi Features for Tactile Sensing: Direct Inference of Pressure, Shear, and Contact Locations

Classification of Household Materials via Spectroscopy

The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies

Material recognition using tactile sensing

Multimodal Tactile Perception of Objects in a Real Home

Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns

ViTac: Feature Sharing Between Vision and Tactile Sensing for Cloth Texture Recognition

From BoW to CNN: Two Decades of Texture Representation for Texture Classification

Recognizing Material Properties from Images

GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force

Active Clothing Material Perception Using Tactile Sensing and Deep Learning

Implementing tactile behaviors using FingerVision

Semi-Supervised Haptic Material Recognition for Robots using Generative Adversarial Networks

Smartphone-Based Food Diagnostic Technologies: A Review

NIR spectroscopic sensing for point-of-need freshness assessment of meat, fish, vegetables and fruits

Shape-independent hardness estimation using deep learning and a GelSight tactile sensor

Hand-Crafted vs Learned Descriptors for Color Texture Classification

Evaluating Deep Convolutional Neural Networks for Material Classification

Yale-CMU-Berkeley dataset for robotic manipulation research

Aggregated Residual Transformations for Deep Neural Networks

Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables

RadarCat: Radar Categorization for Input & Interaction

Fine-Grained Material Classification Using Micro-geometry and Reflectance

Ultra-portable, wireless smartphone spectrometer for rapid, non-destructive testing of fruit ripeness

Densely Connected Convolutional Networks

A 4D Light-Field Dataset and CNN Architectures for Material Recognition

Material Classification Using Raw Time-of-Flight Measurements

On the Performance of GoogLeNet and AlexNet Applied to Sketches

Deep Residual Learning for Image Recognition

Deep learning for tactile understanding from visual and haptic data

Visualizing and Understanding Deep Texture Representations

Multimodal deep learning for robust RGB-D object recognition

Material Recognition from Heat Transfer given Varying Initial Conditions and Short-Duration Contact

Texture Synthesis Using Convolutional Neural Networks

Material recognition in the wild with the Materials in Context Database

Very Deep Convolutional Networks for Large-Scale Image Recognition

Grounding semantic categories in behavioral interactions: Experiments with 100 objects

Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps

Material classification based on thermal properties — A robot and human evaluation

Investigation of a biomimetic fingertip's ability to discriminate fabrics based on surface textures

Sensing and Recognizing Surface Textures Using a GelSight Sensor

Tactile identification of objects using Bayesian exploration

Haptic classification and recognition of objects using a tactile sensing forearm

Identification and Classification of Plastic Resins using Near Infrared Reflectance Spectroscopy

Bayesian Exploration for Intelligent Identification of Textures

Multimodal Deep Learning

Vibrotactile Recognition and Categorization of Surfaces by a Humanoid Robot

Tactile-Data Classification of Contact Materials Using Computational Intelligence

Multimodal fusion for multimedia analysis: a survey

Exploring features in a Bayesian framework for material recognition

Living in a material world: how visual cues to material properties affect the way that we lift objects and perceive their weight.

A review of near infrared spectroscopy and chemometrics in pharmaceutical technologies.

The role of textural statistics vs. outer contours in deep CNN and neural responses to objects

Robotic learning of haptic adjectives through physical interaction

Vision-based material recognition for automated monitoring of construction progress and generating building information modeling from unordered site image collections

Toward Robust Material Recognition for Everyday Objects

NIR and Raman: Complementary Techniques for Raw Material Identification

Infrared and near-infrared technology for the food industry and agricultural uses: on-line applications

is with the Robot Autonomy and Interactive Learning Lab, Georgia Institute of Technology, Atlanta, GA., USA

Field of Study