ParaDocs

Team E Progress Review 7 - Abhishek

Goals Review

1

Our (original) Goals for PR7

• Docker-ize and restructure the repository into a monorepo format
• Conduct a regression test of the entire system to compare against SVD baseline
• Integrate the perception pipeline from Encore
• Define requirements for the UI and develop an initial prototype

​

​

• Wed, Sep 4 - We had sent out these goals
• Fri, Sep 6 - Fall Scope meeting with Dr. Branko + Team Hipster
• Sun, Sep 8 - Internal Design meeting based on feedback

Hence there have been adjustments.

​

Goal: Dockerize System + MonoRepo

• Completed
• Dockerfile + Docker scripts in a dedicated folder in the mono repo
• Entire setup can be duplicated on new machines with no package issues
• Tested both in simulation and on the real KUKA arm

Goal: SVD Regression Test

• Completed
• Verified the system is functional end-to-end with no hardware/software issues
• Tested both with our original setup and dockerized version
• Avg Accuracy is ~ 2mm-4mm over a couple of runs

Goal: UI Prototype

• Has been completely deprioritized based on discussion with our sponsors
• Emphasis on Function over Form + Build components as we need them
• Prototyped and integrated a quick UI for the perception pipeline ( Next Slide )
• We do need a small UI for triggering commands like Re-Register / Execute Plan etc.
• Decided to integrate this into Rviz itself via custom plugin (WIP)

Goal: Perception Pipeline Improvements

• Original Goal : Integrate the perception pipeline from Encore
• But this pipeline
• Is rather over-engineered with many priors and multiple large models
• Quite messy since most code was written in the lead up to encore
• Based on discussion with sponsor, we agreed on getting the initial input prompt from the surgeon
• Faster, More Efficient
• Better Generalization
• Closer to actual scenario in an operating room
• Tradeoff : Less Autonomy
• Our sponsor also emphasized motion tracking and re-registration, so our Encore pipeline wouldn’t have met the requirements anyway

Goal: Perception Pipeline Improvements

• Developed a quick UI to get user inputs from the surgeon
• Integrated with Segment-Anything-2

Registration Priors

Assumptions we make.

•Femur is always on the left

•Orientation < ~45 deg

•Located within crop region

​

•Femur is always on the left

•Orientation < ~45 deg

•Anatomically, femur is located above tibia

​

Assumptions we make in the new pipeline.

Since we are getting user input, we don’t need any of these priors anymore.

​

Additional Progress

• We didn’t have any ( tangible ) additional progress
• Any extra time was spent in design meetings and scoping out the rest of the semester

Additional Progress

2

Registration Technique’s Metric

1. Accuracy: The registration must align the given .stl files with the point cloud captured from the depth camera with a high degree of precision.

​

• Speed: The registration process must be completed within a timeframe that supports efficient surgical workflow.

​

• Scalability: The registration technique must be capable of handling varying sizes and complexities of data sets(which we dont have at all), ensuring consistent performance across different cases with varying degrees of anatomical variability.

Registration Technique’s Metric

Accuracy: Fiducial Localization Error (FLE)

​

The error in determining the location of a point which is used to estimate the transformation.

Registration Technique’s Metric

Medial condyle

Lateral epicondyle

Lateral condyle

Adductor tubercle

Lateral epicondyle

Lateral condyle

Adductor tubercle

Medial condyle

Registration Technique’s Metric

Workflow:

​

Define standard error

Implementation

Test

&

Validation

Keep Improving

Challenges

2

Challenges

• Initial overhead of getting familiar with Docker and restructuring our codebase
• Major blocker and we couldn’t progress on other fronts without it
• Some work we had planned is taking more effort than anticipated
• Custom RViz UI Plugin
• Integrating SAM Perception Pipeline

Risk Management

3

Risks Managed

• Not able to get docker working
• Potential performance / hardware access issues
• Hardware Issues / System Regression since last semester
• Already ordered parts / bones we anticipate that we will need

Potential Risks

• Will still take time for everyone to get used to developing with the new workflow (docker/monorepo) -> Can cause delays
• Some of our plans are conditioned on getting the new system -> If this gets delayed, will have to replan
• Still some unanswered questions that we need to explore
• How will we track any motion efficiently?
• Role of the 2nd camera etc.

Future Work

4

Future Work

• P0 : Analyze and Quantify different sources of error by next sponsor meeting
• P0 : Setup new PC and get registration performance metrics
• P0 : Integrate UI+SAM2 Pipeline completely with MoveIt Stack
• P1 : Explore tracking methods such as FoundationPose / CoTracker
• P1 : Test and evaluate OCS2 based Control
• P2 : Scope out Behavior Tree flow
• P2 : Build pending UI elements

Thank You and Break a Leg!

Please keep this slide for attribution

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon and infographics & images by Freepik