1 of 91

Critical Design Review

Team E - Surgical Robot for Orthopedics

ParaDocs

2 of 91

The ParaDocs are

With Branko, Parker, Sundaram, Kaushik

Paramjit

Shivangi

Li-Wei

Kimi

Abhishek

3 of 91

Project Description

1

4 of 91

Osteoarthritis

5 of 91

Tekkneeca: Robotic Assistant for Knee Replacement

6 of 91

Replace IR Tracking

User’s need	Our solution
IR trackers are invasive and bulky when drilled into patient’s bone	Use RGBD cameras to locate desired drilling site; thus not requiring an IR tracker

7 of 91

Unobstructive system

User’s need	Our solution
IR Camera needs constant Line of Sight; surgeons need to be mindful and work around it.	Design and mount RGBD cameras on the manipulator to avoid interference

8 of 91

Autonomous

User’s need	Our solution
Automate drilling holes on the desired locations on bone	Autonomously plan a trajectory to sites sent by camera, and autonomously drill holes safely

9 of 91

Movement & Obstacles

User’s need	Our solution
There may be movement in the patient’s position, or obstacles in the environment	Dynamic compensation for any motion in the patient, avoid any obstacles in the site

10 of 91

Use Case

2

11 of 91

Shivaknee is suffering from severe pain in her right knee due to osteoarthritis.

Dr. Knee-Wei decides to perform a total right knee arthroplasty, with assistance from Tekkneeca

Use Case

12 of 91

System Requirements

3

13 of 91

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.1 The system shall sense the femur and tibia as a point cloud through a vision-based solution	M.P.1 The system should captured the bones’ point cloud with a density of 0.5 point/mm^3

M.F.2 The system shall sense the surgical workspace as a point cloud	M.P.2.1 The system should captured the surgical workspace point cloud with a density of 1 point/cm^3
	M.P.2.2 The point cloud capture shall focus specifically on the area encompassing the surgical site
M.F.3 The system shall register surgical site with the preoperative bone model continuously in real time	M.P.3.1 The system shall demonstrated a registration accuracy of 1.00 ± 0.5 mm (SOTA:0.68 ± 0.22 mm)
	M.P.3.2 The system shall demonstrated a registration frequency of 5 Hz
M.F.4 The system shall obtain desired drilling pose accurately	M.P.4 The system shall decide the drilling position with 1 mm accuracy and orientation with 1.5°
M.F.5 The system shall obtain desired drilling depth accurately	M.P.5 The system shall decide the appropriate depth for the drilling with 1mm accuracy

14 of 91

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.6 The system shall generate a trajectory to direct the end effector to drilling position	M.P.6 The system shall generate a trajectory insures that the end effector approaches the drill position with an accuracy of 2 mm and an orientation error of less than 2°
M.F.7 The system shall track surgical site in real time throughout the surgical procedure	M.P.7.1 The system shall track bone movement with a latency of less than a second
	M.P.7.2 The system shall track bone movement when displacement exceed 2 mm and orientation exceed 2°
M.F.8 The system shall compensate for any bone motion	M.P.8.1 The system shall compensate motion within 250 milliseconds
	M.P.8.2 The system shall ensure the tooltip remains within 2 ± 0.50 mm of the desired surgical position

15 of 91

Mandatory Functional & Performance Requirements

Functional Requirement	Performance Requirement
M.F.9 The system shall drill holes in the bone	M.P.9 The system shall drill the bones according to the plan with positional accuracy of 2 mm and orientation accuracy of 2°

M.F.10 The system shall visualize the plan	M.P.10 The system shall visualize the surgical plan with latency less or equal to 100 ms
M.F.11 The system shall give basic commands from UI	M.P.11.1 The system's user interface shall respond to command inputs with a latency of less than 100 milliseconds
	M.P.11.2 The system's user interface shall include at least following functions: (1) Start (2) Pause (3) Home
M.F.12 The system shall avoid obstructions in the surgical site	M.P.12 The system shall detect and recognize any obstructions in the surgical site with a reliability of 80%
M.F.13 The system shall stop procedure in-case of emergency	M.P.13 The system shall halt all motions within 100 milliseconds, in the event of an emergency or irregularities

16 of 91

Mandatory Non-Functional Requirements

The system will:

M.N.1 The system will be safe and easy to use for the surgeon
M.N.2 The system will have a procedure time not more than 5-10 mins compared to standard commercial surgery (currently 30-40 mins)
M.N.3 The system will not be an obstruction to the surgeon

17 of 91

Functional Architecture

4

18 of 91

Imagine the system as a well-coordinated orchestra, where every section has a critical role.

We begin with the 'Inputs' — the pre-stored surgical plan, a digital twin of the patient's pre-operative bone model, and real-time data from the patient's bones and the surgical site. With these information, we first do the sensing job here at perception stage, where the robot senses the whole operation site, do the registration and create two point cloud, which is used for calculating the desired drilling position. And to ensure precision , it's designed to compensate for any patient movement in real time.

In the 'Actuation' stage, we see the result of this meticulous planning — the end effector accurately moves to the drilling site and performs its task, creating precise holes in the femur and tibia bones.

Finally, the 'Outputs' of this process are twofold: the physical, with accurately drilled bones ready for the knee implant, and the digital, with a visualized output allowing for real-time monitoring by the surgical team.

Feedback: For Avoid Obstacles, do you assume that the environment does not change?

There should be some functional relationship between Avoid Obstacles and Move EE to Drill Site.

19 of 91

Cyber-physical Architecture

5

20 of 91

21 of 91

Current System Status

22 of 91

System/Subsystem

Descriptions

6

23 of 91

Overall System

Aim: To precisely and robustly register the drilling site, navigate to site and drill holes sufficient to slide in a surgical pin

Autonomous
Non-Invasive
Robust

24 of 91

The subsystems

Perception Subsystem
Manipulation Subsystem
Drilling Subsystem

25 of 91

Perception Subsystem : Overall

26 of 91

Perception Subsystem : SVD Focus

27 of 91

D405 Raw* Point Cloud

And an input image that looks like …

28 of 91

Perception Subsystem: Current Registration Pipeline

Refinement

Local Registration

Methods

Point-to-Plane ICP
CPD/Filterreg
RANSAC+ICP

Extraction

Femur Extraction

Pipeline

Manual Crop
RANSAC Plane Segmentation
DBSCAN Clustering

Estimation

Global Registration

Methods

Open3D FPFH
FMR
Custom Heuristic

29 of 91

I. Manual Crop

III. DBSCAN Clustering

IV. Extracted Bone

II. RANSAC Plane Seg

30 of 91

Registration with ICP

31 of 91

Registration Priors

Assumptions we make.

•Femur is always on the left

•Orientation < ~45 deg

•Located within crop region

32 of 91

Perception Subsystem: New Registration Pipeline

Refinement

Local Registration

Methods

Point-to-Plane ICP
CPD/Filterreg
RANSAC+ICP

Combined Extraction +

Global Registration

Use state-of-the-art foundation models.

Use Grounding DINO to obtain an informed crop.
Get highest point from depth and use that to prompt Segment-Anything.
Get Femur only mask and project to 3D for registration.
Extract orientation for initial transform estimate.

33 of 91

Perception Subsystem: New Registration Pipeline

34 of 91

Perception Subsystem: New Registration Pipeline

Red - STL File (Source)

Dark Blue - Projected SAM Mask (Target)

35 of 91

Perception Subsystem: New Registration Pipeline

36 of 91

Perception Subsystem: New Registration Pipeline

37 of 91

We know where the surgical plan is in 3D space

38 of 91

Perception Subsystem: New Registration Pipeline

39 of 91

Perception Subsystem: New Registration Pipeline

*Detection model fails 50% of the time

40 of 91

Registration Priors

Assumptions we make.

•Femur is always on the left

•Orientation < ~45 deg

•Located within crop region

•Femur is always on the left

•Orientation < ~45 deg

•Anatomically, femur is located above tibia

Assumptions we make in the new pipeline.

41 of 91

Progress Review 1

SVD Encore

42 of 91

Manipulation Subsystem

Description: This subsystem refers to the planning and control stack of the robot. It encompasses everything that goes into getting the end effector to the position designated by the perception pipeline.

Desired state: Drill bit can reach desired global coordinates and execute drilling operations.

43 of 91

We start off with a Surgical Plan

44 of 91

Manipulation subsystem: Testing

45 of 91

Planning and Manipulation: SVD

46 of 91

Drilling Subsystem

Aim: To have an autonomous drill powerful and accurate enough to drill 3mm wide hole(s) at desired location(s) to place surgical pin

Refers to ROS compatible end effector, including drill machine, PCB interface,

camera mount, and drill mount

Drill Choice: Dremel 9100

47 of 91

Drilling subsystem: Models

48 of 91

Drilling subsystem: Electronic Interface

49 of 91

Drilling subsystem: Primary Interim Test on bulb

To prevent inadvertent damage to the (expensive) drill, we tested on light bulb, with simple wiring.

Sent command to Arduino Serial Monitor.

‘Drill’ turned on the drill successfully

‘Stop’ stopped the drill successfully

50 of 91

Drilling subsystem: SVD & SVD Encore Results

51 of 91

Spring Validation Demo

52 of 91

Targeted System Requirements

M.F.1 The system shall sense the femur and tibia as a point cloud through a vision-based solution

M.F.2 The system shall sense the surgical workspace as a point cloud

M.F.3 The system shall register surgical site with the preoperative bone model continuously in real time

M.F.4 The system shall obtain desired drilling pose accurately
M.F.5 The system shall obtain desired drilling depth accurately

M.F.6 The system shall generate a trajectory to direct the end effector to drilling position
M.F.7 The system shall track surgical site in real time throughout the surgical procedure

M.F.8 The system shall compensate for any bone motion

M.F.9 The system shall drill holes in the bone

M.F.10 The system shall visualize the plan
M.F.11 The system shall give basic commands from UI

M.F.12 The system shall avoid obstructions in the surgical site
M.F.13 The system shall stop procedure in-case of emergency

Perception

Planning

Actuation

Surgeon I/O

53 of 91

ParaDocs

Verification Criteria

Did we do it right?

8mm

54 of 91

ParaDocs

Verification Criteria

Did we do it right?

< 8mm

55 of 91

The Results

ParaDocs

Pre-SVD Runs + SVD

56 of 91

Pre-SVD Runs

ParaDocs

Median 4mm | Average 3.9mm

Bone Mount \| With Correction \| April 17
Trial	Result	Error Approx	Arm Speed
1	S	1mm	M
2	S	2mm	M
3	S	4mm	M
4	S	2mm	M
5	S	5mm	M
6	S	3mm	M
7	S	8mm	M
8	S	5mm	M
9	S	4mm	M
10	S	5mm	M

57 of 91

SVD Runs (April 18 2024)

ParaDocs

Spring Validation Demo \| April 18
Trial	Result	Error Approx
1	S	<= 4mm
2	S	<= 4mm
3	Connection Err	-
4	S	<= 4mm

58 of 91

SVD Encore Runs (April 18 2024)

ParaDocs

Encore\| April 25
Trial	Result	Error Approx
1	S	<= 2mm
2	Third Hole Failed	First two holes <=2mm
3	S	All holes within 8mm

59 of 91

60 of 91

ParaDocs

Strengths

Decently robust and accurate perception
Accuracy within 8mm (within 4mm on a good day)
End-to-end integration complete
Hasn’t hurt anyone :) (yet)

61 of 91

ParaDocs

Challenges

and our next steps

Registration doesn’t work 100% of the time and alignment isn’t perfect
Registration isn’t real-time and takes a while
MoveIt2 Planner can be slow and unreliable
No dynamic obstacle avoidance
Open loop ; no feedback
Camera calibration is icky
Spaghetti codebase
I wouldn’t trust my knee with it (yet)

Current Mitigation Strategy

Liberal use of Emergency Stop

62 of 91

Project

Management

7

63 of 91

Updated Work Breakdown Structure

64 of 91

Schedule from PDR

For milestone #3, we are behind schedule about a week.
For milestone #4, we are on track and we believe we can finish on time.

65 of 91

Schedule

We are on track now

66 of 91

Schedule

67 of 91

High-level Test Plan (Spring)

Identifier	Date	Capabilities	Test(s)	Requirement(s)
PR #3	21st Mar.	Drill interfaced with ROS2 Registration working. Can execute registration on live point cloud data Can use MoveIt to control the actual arm while avoiding collisions	T5, T6, T7, T8, T9	M.F.3 M.F.10
PR #4	4th Apr.	Arm can now perform drilling motions Perception pipeline integrated with the manipulation stack Demonstrate E-stop for both drill and arm	T10, T11, T12	M.F.4 M.F.5 M.F.6
PR #5 (SVD)	18th Apr.	Obtain drilling pose and depth from surgical plan through registration Move end-effector to drill pose Drill a hole to planned depth	T13	M.F.9 M.F.13
PR #6 �(SVD Encore)	25th Apr.	Improve the system according to the feedback from SVD	X	X

68 of 91

High-level Test Plan (Fall)

Identifier	Date	Capabilities	Validation
PR #7	Early September	Setup system after summer	All the tests from Spring should run successfully
PR #8	Mid-September	Surgeon I/O, Tibia registration, Replanning for dynamic compensation/obstacles	Arm should stop, replan trajectory when surgeon sends an override from IO Tibia registration achieves expected accuracy (8 mm)
PR #9	Early October	Dynamic compensation with vision	The manipulator should compensate motion using perception subsystem
PR #10	Mid-October	End-to-end autonomous drilling	Subsystem Integration

69 of 91

High-level Test Plan (Fall)

Identifier	Date	Capabilities	Validation
PR #11	Mid-November	Reach performance requirements on drill accuracy	Performance requirements for drill accuracy met Camera publishes accurate location of new drilling site after motion
PR #12 �(FVD)	Late November	Meet all performance requirement	FVD tests, performance requirements
PR #13 �(FVD Encore)	Late November	Meet all performance requirement	FVD tests, performance requirements

70 of 91

FVD

Scope:

Both cameras

Multiple holes

Tibia Registration

Dynamic Comp

FVD

Dynamic Compensation (Sep 16)
Surgeon I/O (Sep 16)
Dynamic Compensation and active registration integrated (Oct 14)
Reach desired accuracy for drill (Nov 08)
Integration and Testing (Nov 18)

71 of 91

FVD

Test Location: B512

Sequence of Events:

The robot begins in a standby state, resting at the designated home position, poised for action.
Once commanded, the arm move end-effector to the precise drilling location.
The drill operation is executed, the robot will drill multiple holes both on femur and tibia
The robotic arm retreats back to the home position
In the process the robot would compensate for the bone motion and still drill accurately

Quantitative Metrics:

Functional / Performance requirements relevant to FVD
Tests relevant to FVD

72 of 91

Budget

Total budget: $5000
Spent: $908.96 (~18% of total budget)
Expect to spend much more on �bone model

73 of 91

Risk Management Table

Risk ID	Risk	Requirement	Type*	Likelihood	Consequence	Mitigation
R1	Cameras are not accurate	M.F.1, M.F.2, M.F.3, M.F.7	T	1	4	- Test cameras immediately upon receiving - Calibrate cameras - Adjust camera mounts - Brainstorm alternate vision solutions
R2	MRSD 3D Printer unavailable	M.F.1, M.F.2, M.F.9	T+S	4	2	- Talk to Tim in the RI shop - Use TechSpark printer - Get a personal 3D printer
R3	Manipulator is damaged	M.F.8, M.F.9, M.F.12	T	1	5	- Follow user manual for safely operating and transporting the arm - Store arm in a secure area away from liquids, weather conditions
R4	Drilling mechanism insufficient	M.F.9	T	2	4	- Purchase drill as soon as semester begins - Test drill on bone model to confirm drill bit and strength - Design a custom drill (a basic literature review has been done for alternatives)

*T: Technical, S: Schedule

74 of 91

Risk Management Table

*T: Technical, S: Schedule

Risk ID	Risk	Requirement	Type*	Likelihood	Consequence	Mitigation
R5	Drill injures a person	M.F.13	T	1	4	- Design and build and test our E-Stop - Follow drill user manual for safe operation - Keep people a safe distance away - Have a first-aid kit ready
R6	A group member is absent	X	S	3	3	- Have a primary and secondary assignee for each task - Check member mental/physical status during stand-up meetings
R7	Version conflicts or data loss	X	T	4	3	- Set up version control software (Git & GitHub) - Backed up blob on Google Drive
R8	Lost connection to the manipulator	M.F.9	T	2	5	- Keep cable taped to the ground - Modify FRI client behavior, override the state

75 of 91

Risk Management Table

*T: Technical, S: Schedule

Risk ID	Risk	Requirement	Type*	Likelihood	Consequence	Mitigation
R9	PCB not functioning	M.F.9	T	3	3	- Prepare backup circuit with same function - Run ERC/DRC to make sure the PCB works
R10	Planning fails	M.F.6	T	3	5	- Replan after planning fail - Define pre-drill pose to avoid singularities

76 of 91

Risk Likelihood-Consequence Table

Before Mitigation

After Mitigation

R1, R2, R4

R2

R3

R4

R5,

R10

R1, R5, R10

R6, R9

R7

R7, R8

R6

R9

R8

77 of 91

Conclusion

8

78 of 91

Conclusions

Key lessons learned during the spring semester.

Integration takes tremendous amounts of time
Project management and teamwork
Progress reviews helped us be on track
It’s crucial to test all cases, and plan fallbacks

Key planned activities during the fall semester.

Bring registration speed from SPF (Seconds Per Frame) to FPS.
Add and calibrate the second camera
Improve our planning stack
Dynamic compensation for movement and obstacles

79 of 91

Thank you and Break a Leg!

80 of 91

Appendix

9

81 of 91

Risk 1 (Inactive)

Risk Title	R1: Cameras are not accurate	Date Submitted	12/03/23
Risk Owner	Kimi	Date Updated	01/31/24
Description		Risk Type
The Realsense D435 / D405 are not providing data accurate enough for registration and compensation		Technical

Consequence		Metrics
We can’t properly register the patient’s bone and guide the manipulator.		Likelihood	Consequence
		1	4
Mitigation Plan
Action		Outcome
- Test cameras immediately upon receiving - Calibrate cameras - Adjust camera mounts - Brainstorm alternate vision solutions		Ensure better registration and reduce likelihood of camera failure.

R1

82 of 91

Risk 2 (Inactive)

Risk Title	R2: MRSD 3D Printer unavailable	Date Submitted	12/03/23
Risk Owner	Li-Wei	Date Updated	01/31/24
Description		Risk Type
The 3D printer in the MRSD room does not function, or is not available due to high demand		Technical + Schedule

Consequence		Metrics
We can’t print the parts for camera mount, end effector, drill mount, test bones		Likelihood	Consequence
		4	2
Mitigation Plan
Action		Outcome
- Talk to Tim in the RI shop - Use TechSpark printer - Get a personal 3D printer		Ensures we have alternative options in case of unavailability

R2

83 of 91

Risk 3 (Mitigated)

Risk Title	R3: Manipulator is damaged	Date Submitted	12/03/23
Risk Owner	Shivangi	Date Updated	04/01/24
Description		Risk Type
The manipulator sustains physical damage during transportation or usage		Technical

Consequence		Metrics
Manipulator does not function properly		Likelihood	Consequence
		1	5
Mitigation Plan
Action		Outcome
- Follow user manual for safely operating and transporting the arm - Store arm in a secure area away from liquids, weather conditions		Reduces likelihood of the manipulator getting damaged

R3

84 of 91

Risk 4 (Inactive)

R4

Risk Title	R4: Drilling mechanism insufficient	Date Submitted	12/03/23
Risk Owner	Param	Date Updated	04/05/24
Description		Risk Type
Off-the-shelf dremel may not be able to interface with our system/ may not meet our requirements		Technical

Consequence		Metrics
Unable to drill holes in the bone		Likelihood	Consequence
		2	4
Mitigation Plan
Action		Outcome
- Purchase drill as soon as semester begins - Test drill on bone model to confirm drill bit and strength - Design a custom drill (a basic literature review has been done for alternatives)		Holes will be drilled in the bones

R4

85 of 91

Risk 5 (Mitigated)

Risk Title	R5: Drill injures a person	Date Submitted	12/03/23
Risk Owner	Abhishek	Date Updated	04/29/23
Description		Risk Type
Accidentally, control to the drill is lost and it injures a person		Technical

Consequence		Metrics
The medical robot does the opposite of its job		Likelihood	Consequence
		1	4
Mitigation Plan
Action		Outcome
- Design and build and test our E-Stop - Follow drill user manual for safe operation - Keep people a safe distance away - Have a first-aid kit ready		Reduces likelihood of injury, and lessens the severity

R5

86 of 91

Risk 6 (Mitigated)

Risk Title	R6: A group member is absent	Date Submitted	02/29/24
Risk Owner	Li-Wei	Date Updated	04/29/24
Description		Risk Type
Team members fall sick, lose motivation to work or have arrangement (e.g. Interview)		Schedule

Consequence		Metrics
Team productivity drops		Likelihood	Consequence
		3	3
Mitigation Plan
Action		Outcome
- Have a primary and secondary assignee for each task - Check member mental/physical status during stand-up meetings		Reduces consequence of team member absent

R6

87 of 91

Risk 7 (Mitigated)

Risk Title	R7: Version conflicts or data loss	Date Submitted	02/29/24
Risk Owner	Li-Wei	Date Updated	03/15/24
Description		Risk Type
Different code modification conflicts, or have no back-up, causing data lost		Technical

Consequence		Metrics
Team progress reduced		Likelihood	Consequence
		4	3
Mitigation Plan
Action		Outcome
- Set up version control software (Git & GitHub) - Backed up blob on Google Drive		Reduces likelihood of version conflict. Although consequence remains serious, the risk is effectively managed

R7

88 of 91

Risk 8 (Active)

Risk Title	R8: Lost connection to the manipulator	Date Submitted	04/01/24
Risk Owner	Li-Wei	Date Updated	04/18/24
Description		Risk Type
FRI connection quality drops below Good, Kuka arm would enter state that only allows monitor but not allows actuation		Technical

Consequence		Metrics
Kuka arm stop moving, FRI client crashes		Likelihood	Consequence
		2	5
Mitigation Plan
Action		Outcome
- Keep cable taped to the ground - Modify FRI client behavior, override the state		Even if the connection drops below Good, the system would wait until the connection is Good and reconnect

R8

89 of 91

Risk 9 (Inactive)

Risk Title	R9: PCB not functioning	Date Submitted	04/08/24
Risk Owner	Shivangi	Date Updated	04/12/24
Description		Risk Type
PCB not able to switch on/off the drill.		Technical

Consequence		Metrics
PCB not able to switch on/off the drill.		Likelihood	Consequence
		3	3
Mitigation Plan
Action		Outcome
- Prepare backup circuit with same function - Run ERC/DRC to make sure the PCB works		The drill can be switched on/off using the PCB

R9

90 of 91

Risk 10 (Active)

Risk Title	R10: Planning fails	Date Submitted	04/25/24
Risk Owner	Paramjit	Date Updated	04/28/24
Description		Risk Type
Planned path exceed joint limits		Technical

Consequence		Metrics
Kuka arm stop moving		Likelihood	Consequence
		3	5
Mitigation Plan
Action		Outcome
- Replan after planning fail - Define pre-drill pose to avoid singularities		Kuka arm can execute the plan successfully

R10

91 of 91

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon and infographics & images by Freepik