1 of 38

Titolo presentazione�sottotitolo

Milano, XX mese 20XX

Master of Science in Biomedical Engineering

Advisor: Prof. Elena De Momi Author: Lorenzo Civati

Co-advisor: Chun-Feng Lai Student ID: 920599

Academic year 2021-2022

Visual servoing control and modeling of a soft robotic endoscope

2 of 38

Clinical background

Lorenzo Civati

2

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Flexible ureterenoscope (FlexXC; Karl Storz, Tuttlingen, Germany)

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drives the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

3 of 38

Clinical background

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

2

Giusti et al.

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Flexible ureterenoscope

(FlexXC; Karl Storz, Tuttlingen, Germany)

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drive the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

4 of 38

Clinical background

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

2

Giusti et al.

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Flexible ureterenoscope

(FlexXC; Karl Storz, Tuttlingen, Germany)

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drive the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

5 of 38

Clinical background

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

2

Giusti et al.

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Flexible ureterenoscope

(FlexXC; Karl Storz, Tuttlingen, Germany)

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drive the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

6 of 38

Clinical background

2

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

Endoscopic view showing stones presence (on the right) and the dusting procedure (on the left) [Steeve Doizi et al. , 2018]

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drive the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

7 of 38

Clinical background

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

2

Giusti et al.

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Flexible ureterenoscope

(FlexXC; Karl Storz, Tuttlingen, Germany)

Urolithiasis, stone formation in kidneys and urinary tract, is a very common disease, in Italy there are 100.000 new cases [Francesco Porpiglia, 2015].
Flexible Ureteroscopy (fURS) is a mininvasive technique among the first-choice procedures for stone removal.
Surgeon manually inserts and drive the endoscope to reach and either remove or dust the stone [Steeve Doizi et al. , 2018].

Nome_Laureando1, Nome_Laureando2

8 of 38

Robotic solution – Atlascope

In this research line the European project ATLAS (AuTonomous intraLuminAl Surgery) is set [Finocchiaro M. et al. ,2022].
An autonomous device might help surgeons in solving tasks like navigation and target tracking.
Continuum robots with a soft deformable structure can prevent tissue damages.

3

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

fURS issues
Radiation exposure
Ergonomic problems
Operative Room organisation
Difficult space orientation
Sub-optimal visibility
Patient movements

Atlascope

AIM

Nome_Laureando1, Nome_Laureando2

9 of 38

State of the Art – Autonomous continuum robots

Lorenzo Civati

4

Boehler Q. et al. , 2020

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

[Boehler Quentin, 2020]

A Robotic Endoscope Automated via Laryngeal Imaging for Tracheal Intubation (REALITI) was realized [Boehler Quentin, 2020].
Autonomous Flexible Endoscope for Minimally Invasive Surgery has been presented [Xin Ma et al., 2019]

[Xin Ma et al., 2019]

Nome_Laureando1, Nome_Laureando2

10 of 38

State of the Art – Autonomous continuum robots

Lorenzo Civati

5

Boehler Q. et al. , 2020

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

External unsensed perturbations cause unpredictable behaviours.
Non idealities like backlash, friction, tendon coupling are present and they result in a non linear behaviour with a hysteresis shape.

Experimental setup (on the left) exploited to obtain the relathionship between the actuator pressure input and the catheter dispacement (on the up-right). In the lower-right figure the dispacement in time is shown [D. Wu, 2020]

Nome_Laureando1, Nome_Laureando2

11 of 38

Aim of the work

Lorenzo Civati

In this work we will:

We will design and test a visual servoing control algorithm.

We will inspect a learning-based model to find a predictive model for Atlascope.

6

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

12 of 38

Atlascope

Lorenzo Civati

Atlascope is a Tendon driven continuum robot: two antagonistically arranged wires are pulled and realeased to bend the soft arm.
The soft arm is a 3D printed helical structure composed by photopolymer resin [C. Culmone et al., 2020].
Two DC motors rotate the pulleys, a stepper motor moves the structure back and forth.
Endoscopic camera is set in an Eye-In-Hand configuration.
The device is characterised by three Degrees of Freedom: two bending DoFs and an insertion an insertion DoF.

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

7

Jorge F. Lazo et al.

Atlascope actuation [Jorge F. Lazo et al.]

Nome_Laureando1, Nome_Laureando2

13 of 38

Image Based Visual Servoing – IBVS controller

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

14 of 38

Image Based Visual Servoing – IBVS controller

ENDOSCOPIC IMAGE

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

ACTUAL

POSITION

DESIRED POSITION

Nome_Laureando1, Nome_Laureando2

15 of 38

Image Based Visual Servoing – IBVS controller

INPUT

OUPUT

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

IMAGE SPACE

Nome_Laureando1, Nome_Laureando2

16 of 38

Image Based Visual Servoing – IBVS controller

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

17 of 38

Image Based Visual Servoing – IBVS controller

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

18 of 38

Image Based Visual Servoing – IBVS controller

INPUT

OUPUT

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

INPUT

Nome_Laureando1, Nome_Laureando2

19 of 38

Image Based Visual Servoing – IBVS controller

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

20 of 38

Image Based Visual Servoing – IBVS controller

INPUT

8

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

OUPUT

Nome_Laureando1, Nome_Laureando2

21 of 38

Model predictive control - Hysteresis Modeling

9

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

video

Motor angle 1 – image position x

Motor angle 2 – image position y

Nome_Laureando1, Nome_Laureando2

22 of 38

Model predictive control - Hysteresis Modeling

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Neural Network

10

Nome_Laureando1, Nome_Laureando2

23 of 38

Model predictive control - Hysteresis Modeling

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Neural Network

10

Nome_Laureando1, Nome_Laureando2

24 of 38

Exprerimental setup

11

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Target

Endoscopic camera

Soft arm

Motors

Nome_Laureando1, Nome_Laureando2

25 of 38

Experiments – Visual servoing controller

Free space

12

Image space

Target positions

Constrained space

Target positions

Image space

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Experiment 1

Target centering in free space

The circle jumps from a position to the an other position every 15 seconds.
The sequence of jumps is repeated 5 times.

Experiment 3

Target centering in constrained space

The circle jumps from a position to the an other position every 15 seconds.
The sequence of jumps is repeated 5 times.

Experiment 2

Target tracking in free space

The circle moves continuously following a trajectory.
This trajectory is repeated 7 times.

Nome_Laureando1, Nome_Laureando2

26 of 38

Experiments – Visual servoing controller

Experiment 1

Target centering in free space

The circle jumps from a position to the an other position every 15 seconds.
The sequence of jumps is repeated 5 times.

Experiment 3

Target centering in constrained space

The circle jumps from a position to the an other position every 15 seconds.
The sequence of jumps is repeated 5 times.

Experiment 2

Target tracking in free space

The circle moves continuously following a trajectory.
This trajectory is repeated 7 times.

12

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

27 of 38

Experiments – Hysteresis modeling

We collect data for model x and model y.
Dataset 1 is obtained by forcing a sinusoidal input with constant amplitude and constant frequency.
Dataset 2 is obtained by forcing a sinusoidal input with varing amplitude and constant frequency.
Dataset 3 is obtained by forcing a sinusoidal input with constant amplitude and varying frequency.
Dataset 4 is obtained by forcing a sinusoidal input with constant amplitude and constant frequency and target in shifted positions.

13

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Data collection for predictive model x

Data collection for predictive model y

Nome_Laureando1, Nome_Laureando2

28 of 38

Metrics

Parameter	Description

14

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

IMAGE SPACE

Target centering (Experiment 1 and 3)

Target tracking (Experiment 2)

Parameter	Description

Nome_Laureando1, Nome_Laureando2

29 of 38

Metrics

14

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Parameter	Description

Target centering (Experiment 1 and 3)

Parameter	Description

Target tracking (Experiment 2)

IMAGE SPACE

Nome_Laureando1, Nome_Laureando2

30 of 38

Results – Controller performances in Free space�Target centering in Experiment 1

15

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Parameter	Description	Mean ± Std
		11.25±9.26 px
		242.24±7.65 px

Time [seconds]

Tracking error in time

Nome_Laureando1, Nome_Laureando2

31 of 38

Results – Controller performances in Free space�Target tracking in Experiment 2

16

Parameter	Description	Mean ± Std
		59.64±4.40 px
		167.31±17.32 px

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Time [seconds]

Target displacement and tracking error in time

[Pixels]

X circle on the monitor

Y circle on the monitor

Nome_Laureando1, Nome_Laureando2

32 of 38

Results – Controller performances Free vs Constrained Space�Comparison between Experiment 1 and Experiment 3

17

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

	Parameter	Description	Mean ± Std
Free space			7.98±4.75 px
Constrained space			15.13±7.27 px

Free space

Constrained space

Nome_Laureando1, Nome_Laureando2

33 of 38

Results – Hysteresis modeling�Model x

18

Dataset		MAE [degrees]
	0.22	0.24	5.02
	0.12	0.26	2.82
	0.07	0.21	1.24
	0.28	0.27	13.23

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Time [seconds]

Absolute prediction error

Nome_Laureando1, Nome_Laureando2

34 of 38

Results – Hysteresis modeling�Model y

19

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Time [seconds]

Absolute prediction error

Dataset		MAE [degrees]
	0.35	0.30	8.51
	0.19	0.31	3.26
	0.16	0.33	0.19
	1.07	0.61	9.50

Nome_Laureando1, Nome_Laureando2

35 of 38

Results – Hysteresis modeling�Model y

20

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Hysteresis curve

Predicted curve

Experimental curve

Dataset		MAE [degrees]
	0.35	0.30	8.51
	0.19	0.31	3.26
	0.16	0.33	0.19
	1.07	0.61	9.50

Nome_Laureando1, Nome_Laureando2

36 of 38

Conclusions

GOALS

We built a visual servoing controller able to track and center a target in the middle of the endoscopic image.
We built 2 neural network models able to forecast the next angle position even in presence of hysteresis.

LIMITS

The controller is characterised by an high overshoot in the centering tasks and faces difficulties in tracking targets when they are moving fast.
The neural network models especially for the model y are affected by the target initial position change and the first implementation of the model in the results to be unstable.

21

Introduction

Aim

State of the Art

Experiment

Methods

Results

Conclusions

Nome_Laureando1, Nome_Laureando2

37 of 38

Titolo presentazione�sottotitolo

Milano, XX mese 20XX

Thank you for your attention

38 of 38

Titolo presentazione�sottotitolo

Milano, XX mese 20XX

Master of Science in Biomedical Engineering

Advisor: Prof. Elena De Momi Author: Lorenzo Civati

Co-advisor: Chun-Feng Lai Student ID: 920599

Academic year 2021-2022

Visual servoing control and modeling of a soft robotic endoscope