1 of 36

Self-supervised Transparent Liquid Segmentation for Robotic Pouring

Gautham Narasimhan¹, Kai Zhang², Ben Eisner¹, Xingyu Lin¹, David Held¹

¹ Carnegie Mellon University, ² University of Notre Dame

Please feel free to interrupt me for any questions!

2 of 36

Narasimhan, Gautham, et al. "Self-supervised Transparent Liquid Segmentation for Robotic Pouring." ICRA 2022

A robotic pouring system for transparent liquids

3 of 36

Outline

Motivation & challenges involved
Previous approaches
Liquid segmentation method
Pouring system
Generalization experiments

4 of 36

Why is detecting transparent liquids important?

Auto stop water dispensers

Robotic waiters

Pouring request

5 of 36

Why is detecting transparent liquids important?

Medical Robots - Control Automation

Robots watering plants - Getty Images

Robotic pouring - Honda Research

Robots that could pour liquids would also enable us to automate tasks such as cooking, pouring medicines into vials in pharmacies, or watering our plants.

In fact, the original idea for our project was to be able to build robots that can pour pizza sauce onto a pizza base in a robotic kitchen setting. However there are many challenges before we can do this, as you can imagine.

Ideally for real world robotics we need to perceive liquids in 3D space. But to in order to see liquids they need to placed inside of a transparent container and perceiving 3D transparent objects is not an easy problem mainly because we aren't able to reliably capture depth information from transparent surfaces.

So for now, we've constrained the problem to liquid detection on a 2D image plane for the current version of the system.

6 of 36

Why is segmenting transparent liquids challenging?

Diverse backgrounds

7 of 36

Why is segmenting transparent liquids challenging?

Diverse backgrounds

Refraction

8 of 36

Diverse backgrounds

Non-stationary liquids

Refraction

Why is segmenting transparent liquids challenging?

9 of 36

Diverse backgrounds

Non-stationary liquids

Diverse Containers

Refraction

Why is segmenting transparent liquids challenging?

10 of 36

Outline

Motivation & challenges involved
Previous approaches
Liquid segmentation method
Pouring system
Generalization experiments

11 of 36

Schenk et al. - Requires thermal cameras

Related work

12 of 36

Schenk et al. - Requires thermal cameras

Yamaguchi et al. - Works only on non-static liquids

Related work

13 of 36

Monroe et al. - Marker based and requires checkboard

Schenk et al. - Requires thermal cameras

Yamaguchi et al. - Works only on non-static liquids

Related work

14 of 36

Outline

Motivation & challenges involved
Previous approaches
Liquid segmentation method
Pouring system
Generalization experiments

15 of 36

15

Dataset of Colored Liquids

Dataset of Transparent Liquids

Can we use Colored Liquid to Learn about Transparent Liquid?

16 of 36

Convert a horse into a zebra

16

Park, Taesung, et al. "Contrastive learning for unpaired image-to-image translation." ECCV 2020.

We take inspiration from contrastive Learning for Unpaired Image Translation also known as CUT which learns an image translation model between a source and a target domain.
In this example here, CUT learns to translate between an unpaired dataset of horse images and another dataset of zebra images. So, it takes as input an image with a horse and learns a translation model for the target zebra domain.

In a successful result, given a specific patch on the output, for example, the generated zebra forehead highlighted in dark blue on the right, one should have a good idea that it came from the horse forehead highlighted in light blue on the let, and not the other parts of the horse or the background vegetation. They achieve this by using a type of contrastive loss function, called InfoNCE loss, which aims to learn an embedding or an encoder that associates corresponding patches to each other, while dissociating them from other non-corresponding patches. To do so, the encoder learns to pay attention to the commonalities between the two domains, such as object parts and shapes, while being invariant to the differences, such as the textures of the animals. The two networks, the generator and encoder, come together to generate an image such that patches can be easily traceable to the input.

This technique of maximizing the mutual information between corresponding patches, in the source and target domains, results in the generated zebra being in the same pose as the horse in the input image. This is a crucial result for us as we would like to this concept to translate between colored liquids and transparent liquids.

17 of 36

Convert a horse into a zebra

17

(1)

(3)

(2)

To briefly describe what CUT does and how we adapt it for our method. The backbone of CUT is a generator that translates an image of the source domain into the target domain. To encourage this translation, three loss terms are used. First, a standard adversarial GAN loss is used, to encourage the generator to output images that are visually similar to those in the target domain, the second and third terms include a patchwise contrastive loss is used to match corresponding input-output patches at a specific location.

In equation 1, G and D, denote the generator and the discriminator respectively, and X and Y denote the source domain and the target domain, respectively. This is the adversarial loss term for the GAN that encourages the output to be visually similar to images from the target domain.

In equation 2, within the patch-wise contrastive loss, G denotes the generator, H denotes a small multi-layer perceptron that generates the stack of features for contrastive learning, and X is the dataset of patches from the source domain. The first multi-layer contrastive patch loss that takes G,H and X as input maximizes the mutual information between the generated embedding vectors between the source and target domain. The second patch loss that takes G, H and Y as input helps regularize the encoder and minimize unnecessary modifications to a source image. It basically behaves as an identity loss function so that the generator is able to reproduce images without making changes in the target domain.

At test time, we typically use the output of the generator G in equation 3. The generator is divided into an encoder G_encoder and a decoder G_decoder. We take an image X from the source domain, pass it through an encoder followed by a decoder to obtain the generator output in the target domain.

18 of 36

18

Dataset of Colored Liquids

Dataset of Transparent Liquids

Can we use Colored Liquid to Learn about Transparent Liquid?

19 of 36

19

Can we use Colored Liquid to Learn about Transparent Liquid?

Colored Liquid

Generated Transparent Liquid

Here, we show some examples of the image translation process:

The patch-wise contrastive loss encourages the object parts in the source image to be in the same location as the parts in the translated transparent liquid image. For example, in the images shown here, the cup, liquid, surface, and even shadows are in the same locations between the image of colored liquid that is input to the generator compared to the synthetic image of transparent liquid that is output by the generator

The primary difference between these images is that the colored liquid has changed to become transparent; the liquid is in the same location as in the input. This property that the liquid is in the same location in the colored liquid input image as in the output transparent liquid image is crucial to the success of our segmentation method. If we assume that the only property that has changed as a result of the translation is the liquid color, then we can directly use the segmentation masks from the colored liquid domain as pseudo ground truth for the generated transparent liquid image.

We collected 15 distinct pouring videos each of green-colored water and clear water. This resulted in having datasets with roughly two thousand images in the both the colored and transparent liquid domains.

20 of 36

Segment Transparent Liquid

To train our segmentation method, we generate a dataset of transparent liquid images by translating from the colored liquid images. In order to obtain ground truth, we use can either use color thresholding or background subtraction on the colored liquid image. Essentially, we take the image of an empty cup and then collect a video of us pouring colored water into the cup. We can then subtract every frame of the video with the image of the empty cup to obtain the ground truth segmentation mask. Using this method we can then easily generate segmentation labels for the synthetic transparent images without requiring human annotations.

We can use this synthetic labeled dataset of generated transparent liquid images and the ground truth from background subtraction to train a transparent liquid segmentation model S using the Binary Cross Entropy loss between the predicted liquid segmentation mask and the pseudo-ground truth.

21 of 36

Input

Segmented image

Results - Image Segmentation

22 of 36

Input�Image

Segmented image

Results - Image Segmentation

23 of 36

Results - Image Segmentation

24 of 36

Outline

Motivation & challenges involved
Previous approaches
Liquid segmentation method
Pouring system
Generalization experiments

25 of 36

Pouring System

26 of 36

Pouring System

And second, a visual post processing stage that converts a liquid segmentation into an estimate of a water height level in a container and a pouring controller which drives the system to reach a target fill level.

For our robotic pouring experiments, we use a Franka Panda 7-DOF robotic arm, with a custom-built end-effector that rigidly secures a source container for pouring. We place a Azure Kinect RGB camera directly in front of the robot’s workspace and also place the target cup directly in front of the camera at a known location. We fill the source cup with transparent liquid, which will be them poured into the target cup to a pre-specified height.

To minimize any potential errors caused by the segmentation model, we first use an off the-shelf method called TransLab to detect the location of the transparent cup in the scene. Next, we crop the image region around the detected cup location. Then, we use our model to segment the transparent liquid pixels from within this crop. We fit a bounding box to the detected water regions and take the ratio between the bounding box height and cup height to be the percentage of water present in the cup.

We start pouring water at a constant rate and then and then stop pouring once the target height has been achieved. We also tilt the cup in the opposite direction of pouring to minimize any residual pouring.

27 of 36

Pouring System

We conduct two sets of real world robotic experiments to evaluate how our perception system performs on a real pouring task. We choose four different initial fill-levels and target fill-levels, and we conduct 20 pouring trials for each fill-level. We measure the level of liquid after we stop pouring and report the average error across each fill-level.

We show that the pouring system has an average RMSE of 0.94% of the cup height, this translates to roughly millimeter level accuracy in the poured water height.

We also attempt to pour colored liquid using background subtraction to obtain the liquid mask. In this case pouring accuracy is worse, with an average error of 3.38% which translated to half a cm. This is most likely due to the background subtraction model not being robust when there are bubbles and other appearance effects in the water.

I can pause for sometime here to take any questions about the pouring system

28 of 36

Pouring from empty to 75%

29 of 36

Pouring from empty to 50%

30 of 36

Pouring from empty to 25%

31 of 36

Pouring from pre-filled cup to 75%

32 of 36

Outline

Motivation & challenges involved
Previous approaches
Liquid segmentation method
Pouring system
Generalization experiments

33 of 36

Segmentation Results - Diverse Backgrounds

34 of 36

Segmentation Results - Diverse Containers

35 of 36

Image translation can be used to generate paired images of colored and transparent liquids in the same pose.

We propose a novel pipeline to segment transparent liquids using only RGB images without manual annotations.

We show our transparent liquid segmentation system working in a

real world pouring setup.

Key takeaways

36 of 36

Image translation can be used to generate paired images of colored and transparent liquids in the same pose.

We propose a novel pipeline to segment transparent liquids using only RGB images without manual annotations.

We show our transparent liquid segmentation system working in a

real world pouring setup.

Key takeaways

Questions?

https://sites.google.com/view/transparentliquidpouring

Acknowledgements

o you know why I might be seeing this error?�

build/forge_meshing/test_results/forge_meshing/MISSING-gtest-test_deflectedHybridizeScanAndMesh.xml: 1 tests, 0 errors, 1 failures, 0 skipped

2:45

I don't have a launch file for the test, but there are other tests like that as well.

Project Website