A new benchmark for group distribution shifts in hand grasp regression for object manipulation. Can meta-learning raise the bar?

Abstract

Understanding hand-object pose with computer vision opens the door to new applications in mixed reality, assisted living or human-robot interaction. Most methods are trained and evaluated on balanced datasets. This is of limited use in real-world applications; how do these methods perform in the wild on unknown objects? We propose a novel benchmark for object group distribution shifts in hand and object pose regression. We then test the hypothesis that meta-learning a baseline pose regression neural network can adapt to these shifts and generalize better to unknown objects. Our results show measurable improvements over the baseline, depending on the amount of prior knowledge. For the task of joint hand-object pose regression, we observe optimization interference for the meta-learner. To address this issue and improve the method further, we provide a comprehensive analysis which should serve as a basis for future work on this benchmark. adaptation (TTA) methods demonstrate distribution in-the-wild in and object segmentation We propose to achieve this on the grasp prediction problem with a meta-learning algorithm, where the to quickly learn new tasks from a few examples at test time 17, 24]. We then evaluate this method on a novel benchmark for group distribution shifts in hand-object pose regression for object grasping. How does the performance of a CNN pose predictor evolve as the test set grasps diverge from the training set? We answer this question and look at the advantages and limitations of meta-learning for this application via experiments and empirical analysis.