1 of 64

Preliminary Design Review

Team E - Surgical Robot for Orthopedics

ParaDocs

2 of 64

The ParaDocs are

Point of Contact: Dr. Branko Jaramaz

Paramjit

Shivangi

Li-Wei

Kimi

Abhishek

3 of 64

Project Description

1

4 of 64

Tekkneeca: Robotic Assistant for Knee Replacement

5 of 64

Replace IR Tracking

User’s need	Our solution
IR trackers are invasive and bulky when drilled into patient’s bone	Use RGBD cameras to locate desired drilling site; thus not requiring an IR tracker

6 of 64

Unobstructive system

User’s need	Our solution
IR Camera needs constant Line of Sight; surgeons need to be mindful and work around it.	Design and mount RGBD cameras on the manipulator to avoid interference

7 of 64

Autonomous

User’s need	Our solution
Automate drilling holes on the desired locations on bone	Autonomously plan a trajectory to sites sent by camera, and autonomously drill holes safely

8 of 64

Movement & Obstacles

User’s need	Our solution
There may be movement in the patient’s position, or obstacles in the environment	Dynamic compensation for any motion in the patient, avoid any obstacles in the site

9 of 64

Use Case

2

10 of 64

Shivaknee is suffering from severe pain in her right knee due to osteoarthritis.

Dr. Knee-Wei decides to perform a total right knee arthroplasty, with assistance from Tekkneeca

Use Case

11 of 64

System Requirements

3

12 of 64

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.1 The system shall sense surgical site as a point cloud through a vision-based solution	M.P.1 The system should captured the bones’ point cloud with a density of 0.5 point/mm^3

M.F.2 The system shall sense the surgical site as a point cloud.	M.P.2.1 The system should captured the surgical site’s point cloud with a density of 1 point/cm^3
	M.P.2.2 The point cloud capture shall focus specifically on the area encompassing the surgical site
M.F.3 The system shall register surgical site with the preoperative bone model continuously in real time	M.P.3.1 The system shall demonstrated a registration accuracy of 1.00 ± 0.5 mm (SOTA:0.68 ± 0.22 mm)
	M.P.3.2 The system shall demonstrated a registration frequency of 5 Hz
M.F.4 The system shall obtain desired drilling pose accurately	M.P.4 The system shall decide the drilling position with 1 mm accuracy and orientation with 1.5°
M.F.5 The system shall obtain desired drilling depth accurately	M.P.5 The system shall decide the appropriate depth for the drilling with 1mm accuracy

13 of 64

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.6 The system shall generate a trajectory to direct the end effector to drilling position	M.P.6 The system shall generate a trajectory insures that the end effector approaches the drill position with an accuracy of 2 mm and an orientation error of less than 2°
M.F.7 The system shall track surgical site in real time throughout the surgical procedure	M.P.7.1 The system shall track bone movement with a latency of less than a second
	M.P.7.2 The system shall track bone movement when displacement exceed 2 mm and orientation exceed 2°
M.F.8 The system shall compensate for any bone motion	M.P.8.1 The system shall compensate motion within 250 milliseconds
	M.P.8.2 The system shall ensure the tooltip remains within 2 ± 0.50 mm of the desired surgical position

14 of 64

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.9 The system shall drill holes in the bone	M.P.9 The system shall drill the bones according to the plan with positional accuracy of 2 mm and orientation accuracy of 2°

M.F.10 The system shall visualize the plan	M.P.10 The system shall visualize the surgical plan with latency less or equal to 100 ms
M.F.11 The system shall give basic commands from UI	M.P.11.1 The system's user interface shall respond to command inputs with a latency of less than 100 milliseconds
	M.P.11.2 The system's user interface shall include at least following functions: (1) Start (2) Pause (3) Home
M.F.12 The system shall avoid obstructions in the surgical site	M.P.12 The system shall detect and recognize any obstructions in the surgical site with a reliability of 80%
M.F.13 The system shall stop procedure in-case of emergency	M.P.13 The system shall halt all motions within 100 milliseconds, in the event of an emergency or irregularities

15 of 64

Mandatory Non-Functional Requirements

The system will:

M.N.1 The system will be safe and easy to use for the surgeon
M.N.2 The system will have a procedure time not more than 5-10 mins compared to standard commercial surgery (currently 30-40 mins)
M.N.3 The system will not be an obstruction to the surgeon

16 of 64

Functional Architecture

4

17 of 64

Imagine the system as a well-coordinated orchestra, where every section has a critical role.

We begin with the 'Inputs' — the pre-stored surgical plan, a digital twin of the patient's pre-operative bone model, and real-time data from the patient's bones and the surgical site. With these information, we first do the sensing job here at perception stage, where the robot senses the whole operation site, do the registration and create two point cloud, which is used for calculating the desired drilling position. And to ensure precision , it's designed to compensate for any patient movement in real time.

In the 'Actuation' stage, we see the result of this meticulous planning — the end effector accurately moves to the drilling site and performs its task, creating precise holes in the femur and tibia bones.

Finally, the 'Outputs' of this process are twofold: the physical, with accurately drilled bones ready for the knee implant, and the digital, with a visualized output allowing for real-time monitoring by the surgical team.

Feedback: For Avoid Obstacles, do you assume that the environment does not change?

There should be some functional relationship between Avoid Obstacles and Move EE to Drill Site.

18 of 64

Cyber-physical Architecture

5

19 of 64

20 of 64

21 of 64

22 of 64

23 of 64

24 of 64

System/Subsystem

Descriptions

6

25 of 64

Subsystems

Drilling Subsystem
Manipulation Subsystem
Perception Subsystem

26 of 64

Drilling Subsystem

Description: This subsystem refers to the ROS compatible end effector, including: Drill machine, PCB interface, camera mount, and drill mount

Desired state: End effector can be controlled programmatically by ROS for drilling.

Drill choice: Considered two form factors:

Traditional dremel
Dremel with motor separate flex shaft for tool

Chose the second option to reduce the weight of the end effector to maintain higher accuracy

27 of 64

Drilling subsystem: Current Status

After 3 iterations….

28 of 64

Drilling subsystem: Current Status

29 of 64

30 of 64

Manipulation Subsystem

Description: This subsystem refers to the planning and control stack of the robot. It encompasses everything that goes into getting the end effector to the position designated by the perception pipeline.

Desired state: Drill bit can reach desired global coordinates and execute drilling operations.

31 of 64

Manipulation Subsystem Current Status

32 of 64

Perception Subsystem : Overall

33 of 64

Perception Subsystem : SVD Focus

34 of 64

D405 Raw* Point Cloud

35 of 64

RGB + D + Ply Dataset

36 of 64

Perception Subsystem: Current Registration Pipeline

Refinement

Local Registration

Methods

Point-to-Plane ICP
CPD/Filterreg
RANSAC+ICP

Extraction

Femur Extraction

Pipeline

Manual Crop
RANSAC Plane Segmentation
DBSCAN Clustering

Estimation

Global Registration

Methods

Open3D FPFH
FMR
Custom Heuristic

37 of 64

I. Manual Crop

III. DBSCAN Clustering

IV. Extracted Bone

II. RANSAC Plane Seg

38 of 64

Adding random Gaussian noise to the initial estimate for each RANSAC trial

RANSAC Initialization for ICP

Red - STL File (Source)

Blue - Filtered Point Cloud (Target)

39 of 64

RANSAC ICP

Red - STL File (Source)

Blue - Filtered Point Cloud (Target)

Registration Results

Consistent Results on 12/16 samples

40 of 64

RANSAC ICP

41 of 64

Perception Subsystem: ROS Pipeline + Hand-Eye Calibration

42 of 64

Project

Management

8

43 of 64

Original Work Breakdown Structure

44 of 64

Focused Work Breakdown Structure

45 of 64

Schedule

For milestone #3, we are behind schedule about a week.
For milestone #4, we are on track and we believe we can finish on time.

46 of 64

Schedule

47 of 64

High-level Test Plan (Spring)

Identifier	Date	Capabilities	Test(s)	Requirement(s)
PR #3	21st Mar.	Drill interfaced with ROS2 Registration working. Can execute registration on live point cloud data Can use MoveIt to control the actual arm while avoiding collisions	T5, T6, T7, T8, T9	M.F.3 M.F.10
PR #4	4th Apr.	Arm can now perform drilling motions Perception pipeline integrated with the manipulation stack Demonstrate E-stop for both drill and arm	T10, T11, T12	M.F.4 M.F.5 M.F.6
PR #5 (SVD)	18th Apr.	Obtain drilling pose and depth from surgical plan through registration Move end-effector to drill pose Drill a hole to planned depth	T13	M.F.9 M.F.13
PR #6 �(SVD Encore)	25th Apr.	Improve the system according to the feedback from SVD	X	X

48 of 64

High-level Test Plan (Fall)

Date	Capabilities	Validation
End of August	Setup system after summer	All the tests from Spring should run successfully
End of September	Surgeon I/O, Tibia registration, Replanning for dynamic compensation/obstacles	Arm should stop, replan trajectory when surgeon sends an override from IO Tibia registration achieves expected accuracy
End of October	Dynamic compensation with vision, reach performance requirements on drill accuracy	Performance requirements for drill accuracy met Camera publishes accurate location of new drilling site after motion
Mid- to End- November (FVD)	End-to-end autonomous drilling	FVD tests, performance requirements

49 of 64

SVD / FVD

Scope:

1 camera (D405)

1 hole

Femur registration

End-to-End flow

SVD

Workspace setup (31 Jan)
End Effector done (28 Feb)
Registration pipeline setup (Mar 15)
Manipulator & Drill controls (Mar 15)
Integration and Testing (April 15)

Scope:

Both cameras

Multiple holes

Tibia Registration

Dynamic Comp

FVD

Dynamic Compensation (Sep 17)
Surgeon I/O (Oct 07)
Reach desired accuracy for drill (Oct 21)
Dynamic Compensation and active registration integrated (Oct 21)
Integration and Testing (November 18)

50 of 64

SVD / FVD

Test Location: B512

Sequence of Events:

The robot begins in a standby state, resting at the designated home position, poised for action.
Once commanded, the arm move end-effector to the precise drilling location.
The drill operation is executed
The robotic arm retreats back to the home position

Quantitative Metrics:

Functional / Performance requirements relevant to SVD / FVD

Tests relevant to SVD / FVD

51 of 64

Budget

Total budget: $5000
Spent: $751.3 (~15% of total budget)
Expect to spend more on bone models

52 of 64

Risk Management Table

Risk ID	Risk	Requirement	Type*	Likelihood	Consequence	Mitigation
R1	Cameras are not accurate	M.F.1, M.F.2, M.F.3, M.F.7	T	1	4	- Test cameras immediately upon receiving - Calibrate cameras - Adjust camera mounts - Brainstorm alternate vision solutions
R2	MRSD 3D Printer unavailable	M.F.1, M.F.2, M.F.9	T+S	4	2	- Talk to Tim in the RI shop - Use TechSpark printer - Get a personal 3D printer
R3	Manipulator is damaged	M.F.8, M.F.9, M.F.12	T	1	5	- Follow user manual for safely operating and transporting the arm - Store arm in a secure area away from liquids, weather conditions
R4	Drilling mechanism insufficient	M.F.9	T	2	4	- Purchase drill as soon as semester begins - Test drill on bone model to confirm drill bit and strength - Design a custom drill (a basic literature review has been done for alternatives)

*T: Technical, S: Schedule

53 of 64

Risk Management Table

*T: Technical, S: Schedule

Risk ID	Risk	Requirement	Type*	Likelihood	Consequence	Mitigation
R5	Drill injures a person	M.F.13	T	1	4	- Design and build and test our E-Stop - Follow drill user manual for safe operation - Keep people a safe distance away - Have a first-aid kit ready
R6	A group member is absent	X	S	3	3	- Have a primary and secondary assignee for each task - Check member mental/physical status during stand-up meetings
R7	Version conflicts or data loss	X	T	4	3	- Set up version control software (Git & GitHub) - Backed up blob on Google Drive

54 of 64

Risk Likelihood-Consequence Table

Before Mitigation

After Mitigation

R1, R2, R4

R2

R3

R4

R5

R1, R5

R6

R7

R6

55 of 64

Thank you and Break a Leg!

56 of 64

Appendix

9

57 of 64

Risk 1

Risk Title	R1: Cameras are not accurate	Date Submitted	12/03/23
Risk Owner	Kimi	Date Updated	-
Description		Risk Type
The Realsense D435 / D405 are not providing data accurate enough for registration and compensation		Technical

Consequence		Metrics
We can’t properly register the patient’s bone and guide the manipulator.		Likelihood	Consequence
		1	4
Mitigation Plan
Action		Outcome
- Test cameras immediately upon receiving - Calibrate cameras - Adjust camera mounts - Brainstorm alternate vision solutions		Ensure better registration and reduce likelihood of camera failure.

R1

58 of 64

Risk 2

Risk Title	R2: MRSD 3D Printer unavailable	Date Submitted	12/03/23
Risk Owner	Li-Wei	Date Updated	-
Description		Risk Type
The 3D printer in the MRSD room does not function, or is not available due to high demand		Technical + Schedule

Consequence		Metrics
We can’t print the parts for camera mount, end effector, drill mount, test bones		Likelihood	Consequence
		4	2
Mitigation Plan
Action		Outcome
- Talk to Tim in the RI shop - Use TechSpark printer - Get a personal 3D printer		Ensures we have alternative options in case of unavailability

R2

59 of 64

Risk 3

Risk Title	R3: Manipulator is damaged	Date Submitted	12/03/23
Risk Owner	Shivangi	Date Updated	-
Description		Risk Type
The manipulator sustains physical damage during transportation or usage		Technical

Consequence		Metrics
Manipulator does not function properly		Likelihood	Consequence
		1	5
Mitigation Plan
Action		Outcome
- Follow user manual for safely operating and transporting the arm - Store arm in a secure area away from liquids, weather conditions		Reduces likelihood of the manipulator getting damaged

R3

60 of 64

Risk 4

R4

Risk Title	R4: Drilling mechanism insufficient	Date Submitted	12/03/23
Risk Owner	Param	Date Updated	-
Description		Risk Type
Off-the-shelf dremel may not be able to interface with our system/ may not meet our requirements		Technical

Consequence		Metrics
Unable to drill holes in the bone		Likelihood	Consequence
		2	4
Mitigation Plan
Action		Outcome
- Purchase drill as soon as semester begins - Test drill on bone model to confirm drill bit and strength - Design a custom drill (a basic literature review has been done for alternatives)		Holes will be drilled in the bones

R4

61 of 64

Risk 5

Risk Title	R5: Drill injures a person	Date Submitted	12/03/23
Risk Owner	Abhishek	Date Updated	-
Description		Risk Type
Accidentally, control to the drill is lost and it injures a person		Technical

Consequence		Metrics
The medical robot does the opposite of its job		Likelihood	Consequence
		1	4
Mitigation Plan
Action		Outcome
- Design and build and test our E-Stop - Follow drill user manual for safe operation - Keep people a safe distance away - Have a first-aid kit ready		Reduces likelihood of injury, and lessens the severity

R5

62 of 64

Risk 6

Risk Title	R6: A group member is absent	Date Submitted	02/29/24
Risk Owner	Li-Wei	Date Updated	-
Description		Risk Type
Team members fall sick, lose motivation to work or have arrangement (e.g. Interview)		Schedule

Consequence		Metrics
Team productivity drops		Likelihood	Consequence
		3	3
Mitigation Plan
Action		Outcome
- Have a primary and secondary assignee for each task - Check member mental/physical status during stand-up meetings		Reduces consequence of team member absent

R6

63 of 64

Risk 7

Risk Title	R7: Version conflicts or data loss	Date Submitted	02/29/24
Risk Owner	Li-Wei	Date Updated	-
Description		Risk Type
Different code modification conflicts, or have no back-up, causing data lost		Technical

Consequence		Metrics
Team progress reduced		Likelihood	Consequence
		4	3
Mitigation Plan
Action		Outcome
- Set up version control software (Git & GitHub) - Backed up blob on Google Drive		Reduces likelihood of version conflict. Although consequence remains serious, the risk is effectively managed.

R7

64 of 64

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon and infographics & images by Freepik