System Development Review

Team E - Surgical Robot for Orthopedics

ParaDocs

The ParaDocs are

With Branko, Parker, Sundaram, Kaushik

Paramjit

Shivangi

Li-Wei

Kimi

Abhishek

Overview

1

The Project

System to assist in

Total Knee Replacement Surgery

​

Aim: To precisely and robustly register the drilling site, navigate to site and drill holes sufficient to slide in a surgical pin

​

• Autonomous
• Non-Invasive
• Robust

The Project

• Key Components
• KUKA LBR Manipulator
• RealSense D405 and D435 Camera
• Dremel 9100 Drill
• General Workflow
• Register (Where is the bone located in the scene)
• Plan (A collision free path for the end effector)
• Drill (Make hole(s) at designated spots)

​

Use Case

Surgeon

I/O

Main Robot

Manipulation

Subsystem

Patient

Perception Subsystem

Requirements

Modifications

2

Our functional requirements have stayed the same, however the importance of some requirements have changed

​

​

​

Overall Approach

SVD/Encore

End-to-End Pipeline

Systems Focus

• User Interface
• Both femur + tibia
• Drilling multiple points in succession

etc

FVD/Encore

Goal : 2mm / 2°

Overall Approach

SVD/Encore

End-to-End Pipeline

Systems Focus

Algorithmic Focus

• User Interface
• Both femur + tibia
• Drilling multiple points in succession

etc

• Motion Tracking
• Dynamic Compensation
• Registration Performance

etc

FVD/Encore

Goal : 2mm / 2°

Current

System Status

3

Subsystems

Last semester, we had three subsystems we were actively working on,

• Perception
• Drilling
• Manipulation

​

We have finished developing the drill subsystem and our manipulation work is focused on planning hence this semester we have the following primary subsystems,

• Perception: Registration | Tracking
• Planning: Motion Compensation | Obstacle Avoidance

With additional work towards optimization and testing.

Perception: Registration

• We had developed an GroundingDINO (Open-Vocab Object Detection Model) + Segment-Anything Model (SAM) based registration pipeline for SVD Encore
• But this pipeline
• Is rather over-engineered with many priors and multiple large models
• Quite messy since most code was written in the lead up to encore
• Based on discussion with sponsor, we agreed on getting the initial input prompt from the surgeon, instead of DINO
• Faster, More Efficient
• Better Generalization
• Closer to actual scenario in an operating room
• Tradeoff : Less Autonomy

Registration

• Developed a simple UI to get user inputs from the surgeon asking them to annotate points on the femur and tibia
• We give these points as +ve and -ve prompts to SAM2 which enables us to obtain a precise mask.

​

Additionally,

• We use the the annotated femur point as the initial position estimate.
• We use the angle b/w these points as the orientation estimate.

​

Effectively solving the problem of Global Registration for our use case, without needing additional cropping/clustering/segmentation steps. We later perform ICP as usual.

Registration Priors

Assumptions we made for SVD

•Femur is always on the left

•Orientation < ~45 deg

•Located within crop region

​

•Femur is always on the left

•Orientation < ~45 deg

•Anatomically, femur is located above tibia

(For SAM Input)

​

Assumptions we made for SVD Encore

Since we are getting user input, we don’t need any of these priors anymore.

​

Registration

This new pipeline has been integrated with the manipulation stack, effectively replacing our old registration pipeline.

​

Also a testament to our improved software development workflow. Was developed entirely offline with Docker + ROS bag files and worked perfectly the first time with the robot-in-loop.

​

Registration : Analysis

Performance Benchmark

• SAM Mask Generation : 40ms (small) | 140ms (large)
• Unprojection + Preprocessing : 200ms
• ICP : 400ms - 1.2s (depending on RANSAC trials)

​

Total Time - 0.7 - 1.5s

​

​

​

Accuracy Benchmark

• Any evaluation with the actual camera / robot-in-loop will have errors due to calibration, execution etc.
• Hence to get an isolation registration accuracy we are trying to replicate our setup in simulation but currently WIP.

​

​

Perception: Tracking

Tested Meta’s SOTA model : Co-Tracker

​

Currently we are working on figuring out

• Which camera feed to use
• Since this is a huge model, how to improve performance
• How will it interplay with the registration pipeline

​

Planning

Along with perception, this is the primary area of focus.

​

Last semester,

• We were using RRT* via Moveit/OMPL with 10s to optimize.
• Our stack was open-loop. We plan once and then execute it.
• No fallback behavior if a plan fails.
• No obstacle avoidance.

Essentially a “pray and hope for the best” approach.

​

Hence we want to make the following improvements -

• Plan paths with certain constraints (eg - always look at the bone while moving).
• Integrate with tracking system for motion compensation.
• Be able to avoid obstacles.
• Minimize planning time to make the system more reactive.

​

​

​

​

Hybrid-Planning

Instead of the typical “Sense-Plan-Act” loop, hybrid planning enables interleaving global and local planners to adapt to dynamic environments.

• Global Planner
• Running at a lower frequency
• When large motion detected, replan the path
• Plan according to static obstacles
• Local Planner
• Running at a higher frequency
• Execute Global Planner’s plan
• Compensate for small motion

Hybrid-Planning

Hybrid-Planning

Planning Future Work

• Hybrid-planning Manager
• Define logic between and track states.
• Global Planner
• Set planning space limit (virtual walls)
• (Main Challenge) Add constraint to keep looking at the bone
• (Main Challenge) Add D435 obstacles to the planning scene
• Crop out bone and robot arm in the occupancy grid
• Local Planner
• Modify trajopt plugin to take in new goals and do motion compensation
• When compensating for motion, keep looking at the bone

Subsystem Evaluation / Error Quantification

• End-to-end error: roughly <= 4mm (with strict priors)

Project management

3

Schedule

Schedule

We are behind schedule

• Various Error Source Quantified
• Execution Error
• Drill URDF Error
• Camera Calibration Error
• Registration Error
• Real-time Motion Compensation Tested
• Behavior Tree Developed
• Deprioritized
• Bone Tracking Algorithm Developed
• 50% progress

Schedule

Why we are behind schedule

• We spent some time to improve our software setup / workflow
• We spent some time to eval/quantify different sources of error
• We are trying new unfamiliar methods

Our plan to catch up

• Deprioritize non-essential tasks
• Increase time allocation for critical tasks
• Pair programming for important tasks that could be potential bottlenecks.

Test Plan - Progress Reviews

Test Plan - FVD/FVD Encore

• Location: NSH B512
• Equipment:
• KUKA LBR Med7 with drill, Realsense D405 mounted on end-effector, Realsense D435 mounted on workspace, Team E PC, Sawbones bone models, Bone model mount, Surgical guide pins
• Procedure:
• Setup
• The bone is fixed on the mount, within the workspace of the arm
• The arm is in a configuration such that the bone is in the view of D405
• Use a sharpie to mark the desired drilling point on the bone, as outlined in the surgical plan
• The E-stops are kept ready within reach of team member’s hand

Test Plan - FVD/FVD Encore

• Procedure:
• Test
• Run registration pipeline to obtain target pin positions and angles
• The robot should then generate and execute a trajectory to drill multiple holes (at least one each in femur and tibia). The drill should also be triggered autonomously
• Before the first hole is drilled, we will slightly move the bone setup mid-execution to demonstrate that the robot is able to compensate for motion
• If bone is out of sight, the robot should stop execution as a safety feature
• Once the robot finishes drilling, it will return to the home position

Test Plan - FVD/FVD Encore

Test Plan - FVD/FVD Encore

• Quantitative Metrics:
• FVD
• The robot should autonomously drill holes on the femur and the tibia, at the target position without colliding into any static obstacle, and compensating for bone motion
• The hole should be within 2 mm of the marked position and within 2 deg of the target orientation to fit the surgical guide pin
• We will compare the drilled holes with a 3D-printed stencil to verify that it meets the position / orientation error criteria
• FVD Encore
• We will put cutting guides on the bone through surgical pins to demonstrate the accuracy and consistency across multiple holes

Budget

• Total budget: $5000
• Spent:
• $1,390.97 (~27.8%)
• Budget Remaining for emergencies:
• $3,609.03 (~72.2%)
• Primary critical part:
• Synthetic Bones, have 6 in inventory

Risks

*T: Technical, S: Schedule

Risks

*T: Technical, S: Schedule

Risks

*T: Technical, S: Schedule

Risk Likelihood-Consequence Table

R3

R8, R10

R5, R14

R7

R6

R11, R13

R12

Thank You!

Questions?