Distilled Feature Fields Enable Few-Shot Language-Guided Manipulation
Interdisciplinary Research Achievement �Presents a few-shot learning method for 6-DOF grasping and placing that harnesses strong spatial and semantic priors to achieve in-the-wild generalization to unseen objects. Features distilled from a vision-language model, CLIP, provide a way to designate novel objects for manipulation via free-text natural language and generalization to unseen expressions and novel categories of objects. This work won the Best Paper Award at the Conference on Robot Learning (CoRL) in November 2023.
Impact on Artificial Intelligence
Self-supervised and language-supervised image models contain rich knowledge of the world that is important for generalization. Many robotic tasks, however, require a detailed understanding of 3D geometry, which is often lacking in 2D image features. This work bridges this 2D-to-3D gap for robotic manipulation by leveraging distilled feature fields to combine accurate 3D geometry with rich semantics from 2D foundation models.
Impact on Fundamental Interactions
Geometric understanding has been a long-standing problem in computer vision and is an essential part of mapping, grasping, and legged locomotion. This work incorporates pre-trained vision foundation models to augment geometry with semantics, underscoring the importance of the fundamentals of the physical world in developing these advances in robotics.
Outlook
While the success rate of the robot is far from practical for industrial use, the overall strategy of using 2D visual priors for 3D scene understanding can leverage the rapid advancements in VLMs, which hold significant potential for improving performance and generalization.
Ge Yang* (IAIFI Fellow), William Shen*, Alan Yu, Jansen Wong, Leslie Pack Kaelbling, Phillip Isola (MIT)
The NSF Institute for Artificial Intelligence and Fundamental Interactions (IAIFI) is �supported by National Science Foundation under Cooperative Agreement PHY-2019786
Distilled Feature Fields Enable Open-Ended Manipulation. (1) Robot uses a selfie stick to scan RGB images. (2) Extract patch-level dense features for the images from a 2D foundation model, and distill them into a feature field (PCA shown) along with modeling a NeRF. (3) We query CLIP feature fields with language to generate heatmaps and infer 6-DOF grasps on novel objects given only ten demonstrations.
References
[1] https://f3rm.csail.mit.edu [2] https://arxiv.org/abs/2308.07931
Language-Guided Manipulation. (Top Row) Heatmaps given the language queries. (Bottom Row) Robot executing grasps sequentially without rescanning. These objects have never been seen before by the robot.