1 of 33

Supervisor: Priyanka Rao

Yasmeen Hmaidan

1

camera-based shape sensing and motion capturing of tendon-driven continuum robots

2 of 33

2

|

distance

proximity

object

3 of 33

3

|

4 of 33

4

|

5 of 33

5

|

Where am I?

6 of 33

6

|

Oh, I’m here at (x, y, z)!

7 of 33

Table of Contents

1

3

4

2

Shape Sensing

  • Purpose
  • Different Strategies
  • Pros/Cons

Intro to TDCRs

  • Continuum Robot
  • TDCR elements

​

Depth Estimation (methodology)

  • Multiple-Camera system
  • Computer Vision

Project Timeline

  • Workflow
  • Target Outcomes

7

|

8 of 33

Intro to TDCRs

1

Tendon Driven Continuum Robots

8

|

9 of 33

What is a Continuum Robot (CR)?

According to the Burgner-Kahrs, Rucker, & choset, 2015 definition:

​

  • Actuatable structure
  • Constitutive material forms curves
  • Continuous tangent vectors

9

Note, no assumptions are made on:

  • CR’s actuation method
  • Elasticity of composing materials

Emphasis: Continuous curve morphology

​

Pro: conformity

Con: less precise positioning (w/o discrete rigid links)

|

(CSC476, 2020)

10 of 33

CR Elements

​

​

10

Ultimate Goal:

  • Fully controllable continuously bending manipulator

​

Motion Primitives Set:

  • Extension & Contraction
  • Bending
  • Twisting

range of motion ∝ # of stacked segments

Extrinsic Actuators:

  • Tendon/Wire-Actuated
  • Telescoping Pre-Curved Tubes

|

(CSC476, 2020)

11 of 33

Shape Sensing

2

Purpose & Strategies

11

|

12 of 33

Shape Sensing Types

12

Embedded Sensors

  • Fibre-optic sensing
    • Geometric fibres and sensor array
    • Pro: Real-time shape information
    • Con: Expensive
  • Electromagnetic sensing
    • Locates objects with sensor coils
    • Real-time pose tracking
    • Pro: No line-of-sight restrictions
    • Con: Robot rigidity

External Sensors

  • Mechanical probe
    • Touches robot to measure tip position or shape
    • Pro: Measure multiple distinct locations
    • Con: Not contactless

​

​

  • Laser probe
    • Emits laser line that uses images from the camera along robot to get object distance
    • Pro: Dense point cloud of robot shape
    • Con: Time-consuming

|

(CSC476, 2020)

13 of 33

Shape Sensing Types

13

Ideal Choice: External image-based sensing

​

  • Multiple external cameras
  • Contactless
  • Used in both static and dynamic TDCR applications
  • Precise 3D reconstruction
  • Challenge: Direct line-of-sight (occlusions/partial views are processed)

​

|

(CSC476, 2020)

What do I look like?

14 of 33

Depth Estimation (methodology)

3

Multiple-Camera system & Computer Vision

14

|

15 of 33

Multiple-Camera system

15

Main Project Goal:

  • Orient three cameras
  • Calculate 3D transformations to 'global' coordinates
  • Input:
    • Pixel coordinates of detected object (per camera)
  • Output:
    • 3D coordinates in decided frame + the three axes (x,y,z)

|

(Dalvand, 2016)

(Oliveria, 2008)

16 of 33

Camera Calibration Algorithm

16

  • Calculates camera matrix (to detect object’s global location, size, etc.)
    • Two parameters

|

(Mathworks, 2021)

17 of 33

Camera Calibration: ArUco Module Markers

A type of QR code:

  • Black border = fast image detection
  • Inner binary matrix = identifier and detects + corrects distortion error
  • Output: returns a list of detected markers

​

17

|

(OpenCV, 2019)

18 of 33

Calibration Challenges

18

Radial Distortion: straight lines appear curved.

|

Problem: camera distorts images when not parallel to the imaging plane.

Tangential Distortion: some areas in the image look nearer.

Pincushion Distortion

No Distortion

(OpenCV, 2019)

19 of 33

Calibration Challenges:

Stereovision System

19

|

Correct 2D pose estimation by using epipolar relationships.

Triangulated point not synced and not accurate 3D position estimate.

Correct camera locations by optimized orientation & location.

3D reconstruction accuracy of object: ∝ # of cameras and triangulated angles covered.

(DeepFly3D, 2019)

20 of 33

What does good camera calibration look like?

20

Good Camera Calibration = accurate estimates of objects in the world and where the TDCR is in this environment with no blind spots.

​

  • External image-based sensing
  • Three-camera system
  • Intrinsic and extrinsic camera calibration parameters
    • ArUco Markers
  • OpenCV distortion & calibration optimization methods

|

21 of 33

21

|

So this is good enough?

22 of 33

Research Plan

22

​

Data

Application

Goal

ArUco Markers

First 10 markers of ARUCO_MIP_36h12

Rectangle robot base setup

Establish global coordinate system

Camera Setup

Live video feed to image collection

3 cameras in tandem orientation

Calculate extrinsic & intrinsic params

Computer Vision

OpenCV

Image processing pipeline

3D Reconstruction and transformation

|

23 of 33

TDCR Image Processing Pipeline

3D OpenCV Reconstruction:

  • Segmentation
  • Epipolar geometry
  • Distortion reduction

​

23

|

(CSC476, 2020)

(OpenCV, 2019)

24 of 33

Project Timeline

4

Workflow & Target Outcomes

24

|

So what now?

25 of 33

weeks 3-5

Camera calibration

+ Aruco markers

weeks 1-2

OpenCV Tutorials

+ CSC476

weeks 8-10

3D depth mapping + real-time tracking

weeks 6-8

Set up camera system + 3D transforms

weeks 11-12

Extra documentation + report writing

weeks 13-16

TDCR robot trial + GUI + ML joint to tip position mapping

25

|

26 of 33

26

|

Thanks for finding me, CRL!

27 of 33

References

https://www.researchgate.net/figure/Three-wheel-Omnidirectional-robot_fig10_256089781

​

https://april.eecs.umich.edu/software/apriltag.html

​

https://www.researchgate.net/figure/The-DeepFly3D-pose-estimation-pipeline-A-Data-acquisition-from-the-multi-camera_fig4_336273571

​

https://www.researchgate.net/figure/Three-wheel-Omnidirectional-robot_fig10_256089781

​

https://robotics.stackexchange.com/questions/19901/apriltag-vs-aruco-markers

​

https://www.mathworks.com/help/vision/ug/camera-calibration.html

​

https://www.youtube.com/watch?v=E9ka_2mAXvw

​

https://docs.opencv.org/master/da/d13/tutorial_aruco_calibration.html

​

http://biorobotics.harvard.edu/pubs/2016/ref_conf/MDalvand_SMC2016.pdf

27

28 of 33

Calibration Types

28

Intrinsic Calibration

  • Given a point p = (x, y, z) in camera frame
  • Image coordinates calculated from top of frame in a live camera feed
  • w.r.t camera center (principal point) and distance to image plane (focal length)
  • PPM maps p to image coordinates (u,v)
  • Goal: camera parameters

​

​

​

Perspective Projection Model (PPM)

|

(Spartan Robotics, 2020)

29 of 33

Components

29

Extrinsic Calibration

  • Maps where object is in camera frame
  • Then, maps it to robot frame with rotational translation
  • Goal: know where cameras are relative to TDCR in real-time

|

(Spartan Robotics, 2020)

30 of 33

Pros & Cons

ArUco Markers

Pros

​

  • Easier OpenCV implementation
  • Available arUco marker generator
  • Fewer false detection (w/ default)

Cons

  • More computationally intensive
  • Newer versions are GPL licensed, so opencv is older
  • More tuning parameters

AprilTags

Pros

  • BSD License
  • Fewer tuning parameters
  • Long distance compatibility
  • More flexible marker design
  • Less computation

Cons

  • No opencv implementation
  • More steps to obtain markers
  • More false detection (with default parameters)

​

30

|

(OpenCV, 2019)

31 of 33

31

|

ChAruco boards > ArUco boards for camera calibration = more accurate marker corners.

​

Benefits: occlusions and partial views are allowed, and not all the corners need to be visible in all the viewpoints.

(OpenCV, 2019)

32 of 33

32

|

(OpenCV, 2019)

33 of 33

some CRL art!

33

thanks &