1 of 14

Vision-based Pick-n-Place using THORMANG Whole-Body Manipulation.

Final Project Presentation

​

Kazuki Shin, Ali Albazroun, Qingxuan Guo, Huai-Te Wang

Fall 2021

2 of 14

GRAINGER ENGINEERING

Can we automate household tasks using a humanoid robot?

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Original goal task:

  1. walk to kitchen sink
  2. grab dishes from kitchen sink
  3. place dishes into dishwasher

​

Main challenges:

  • kitchen height
    • stair climbing gait
  • sink depth/occlusion
    • advanced perception system
    • motion planning with obstacle avoidance

​

Simplified goal task:

  1. dynamically detect cup position on table
  2. grab cup from table
  3. place into dishwasher

3 of 14

GRAINGER ENGINEERING

Vision - Object tracking using OpenCV

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

RGB to Grayscale

Thresholding

Floodfill

Binary Inversion

Addition

Opening

Largest Object

4 of 14

GRAINGER ENGINEERING

Vision - Object tracking using OpenCV

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

5 of 14

GRAINGER ENGINEERING

Bridge vision to gripper control - pixel to world transformation

​

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Retrieved from

Robert Collins

CSE486 , Penn State

Lecture 12

World Coordinate

Pixel Coordinate

Camera Coordinate

Coordinates Setup:

Film Coordinate

1

3

4

2

4

2

3

1

6 of 14

GRAINGER ENGINEERING

Bridge vision to gripper control - pixel to world transformation

​

Transformation

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Intrinsic Matrix

(optical center,scaling)

Extrinsic Matrix

(camera rotation

and translation)

v

u

1280 x 720 pixels

Image captured by camera on Thormang

Fix to zero

7 of 14

GRAINGER ENGINEERING

Bridge vision to gripper control - pixel to world transformation

​

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Z

V

U

Y

X

W

44 cm

Extrinsic Matrix:

45°

8 of 14

GRAINGER ENGINEERING

Vision - Pixel to World Transformation

​

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Intrinsic Matrix:

By using ROS camera_calibration function, we can acquire

However, the world coordinate values we acquire from the pixel coordinate are not accurate. It may be due to the distortion.

9 of 14

GRAINGER ENGINEERING

Manipulator Control Pipeline

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Control Command

Dynamixel SDK

Motor ID

10 of 14

GRAINGER ENGINEERING

Simulation

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

11 of 14

GRAINGER ENGINEERING

Demo Video

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

12 of 14

GRAINGER ENGINEERING

Walking Module - analysis through ZMP and CoM plots

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

13 of 14

GRAINGER ENGINEERING

Verifying effects of balance algorithm through IMU data collection

ECE598JK Introduction to Humanoid Robotics | Department of Electrical and Computer Engineering

Balance Stabilizer Enabled

No Balance Stabilizer

14 of 14

Questions?