1 of 1

Self-supervised Transparent Liquid Segmentation for Robotic Pouring

Gautham Narasimhan1, Kai Zhang2, Ben Eisner1, Xingyu Lin1, David Held1

Carnegie Mellon University, 2 University of Notre Dame

​

Motivation

Liquid state estimation is important for robotics tasks such as pouring; however, estimating the state of transparent liquids is a challenging problem. We propose a novel segmentation pipeline that can segment transparent liquids such as water from a static, RGB image without requiring any manual annotations or heating of the liquid for training. Our contributions are:

​

  • A novel segmentation pipeline to detect transparent liquids such as water from a static, RGB image without requiring any manual annotations or multi-modal inputs.
  • Utilization of transparent liquid segmentation for a robotic pouring task that controls pouring by perceiving liquid height in a transparent cup.

References

[1] Connor Schenck and Dieter Fox. Visual closed-loop

control for pouring liquids. ICRA 2017

​

[2] Enze Xie, Wenjia Wang, Wenhai Wang,

Mingyu Ding and Ping Luo. Segmenting

transparent objects in the wild. ECCV 2020

​

Fig 6: Robotic Pouring System

Fig 4: Generalization to diverse backgrounds

Fig 5: Generalization to unseen transparent containers

Segmentation Generalization

​

How To Pour transparent liquids?

​

Fig 3: Image translation using unpaired images of colored and transparent water

Fig 1: Overview of our method

Our pipeline for detecting and pouring transparent liquids proceeds as follows:

​

  1. Collect two datasets (unpaired) of colored and transparent liquid in containers
  2. Create synthetic segmentation labels for transparent liquids using image translation
  3. Train a transparent liquid segmentation model using the generated labels
  4. Closed-loop robotic pouring of a specific amount of liquid using our transparent liquid segmentation.

Transparent Liquid Detection

​

Table 1: Pouring performance for various target liquid heights

Method

Table 2: Segmentation performance (IOU) for various target liquid heights

Image Translation:

​

  • We learn to translate colored liquid to transparent liquid.
  • A patchwise contrastive loss maximizes mutual information between liquid patches in both domains.
  • An identity loss is used to minimize modifications to images from source domain.
  • This method requires that the liquid be present in the same region in source and target domains.

Transparent liquid Segmentation:

​

  • We use background subtraction to obtain ground truth labels for segmentation in the colored liquid domain.
  • A Fully Convolutional Network (U-Net) is trained to segment transparent liquids using paired ground truth from background subtraction and image translation
  • Our segmentation pipeline is able to run at roughly 10Hz.
  • We observe that the generated transparent liquid is in the same position as the colored liquid
  • The refraction patterns from the source domain are being maintained.
  • We show generalization to unseen backgrounds from YouTube
  • Diverse background dataset is collected using electronic display panels.
  • We approximate the water column in 2D by fitting a bounding box.
  • Robot pours at a constant rate until target height is achieved.
  • Feedback loop starts after some pouring has occurred for controller stability.

Liquid Domain Translation

  • We show generalization to unseen containers.
  • Segmentation performance drops when glass is nearly full