1 of 38

paraDocs

Progress Review 1

2 of 38

Point of Contact: Dr. Branko Jaramaz

Paramjit

Shivangi

Li-Wei

Kimi

Abhishek

3 of 38

Orthopedic surgeon needs to perform total knee replacement on a patient suffering from severe pain due to osteoarthritis in the right knee.

Our System: Tekkneeca

A robotic system designed to assist a surgeon with total knee replacement surgeries by autonomously drilling into the patient’s bone.

Use Case

4 of 38

Tekkneeca

5 of 38

Our Goals for PR1

Show 3D printed preliminary end-effector design.
Initial registration attempt. (T1)
Gazebo/MoveIt setup and KUKA in simulation. (T2)
Bone mount setup in flexed position.
End-to-end teleoperated drilling of holes into bone. (T3)

M.F.1 - The system shall sense surgical site as a point cloud through a vision-based solution.

6 of 38

Progress Review 1

1

7 of 38

Goal: Show 3D printed preliminary end-effector design

Figure: CAD model of the first iteration of the end effector.

People: Paramjit

8 of 38

Realizations

The camera should be moved ahead since most of it is being blocked by the drill.
3d printed part would mostly not hold up over a long time, and we should make it with metal eventually.

People: Paramjit

Figure: Screenshot of the camera feed from the mounted D405

Clip: Video showcasing the end-effector mounted on Kuka arm

9 of 38

Goal: Initial registration attempt

Initial attempt to register the bone STL to the point-cloud.

Point-cloud data obtained from dummy setup using D405.

People: Abhishek, Kimi

10 of 38

Realizations

ICP may not suffice for robust registration

Started looking at other methods of registration
Talked to HIPSTER - In their case, their point clouds were obtained via the probe which were precise and had way less points. They suggested removing artifacts and tuning the params.

People: Abhishek, Kimi

11 of 38

People: Abhishek, Kimi

Manual Crop

12 of 38

Manual Crop

RANSAC Plane Segmentation

Clustering

13 of 38

People: Abhishek, Kimi

RANSAC Plane Segmentation

Clustering

14 of 38

People: Abhishek, Kimi

Extracted Bone Point Cloud

15 of 38

People: Abhishek, Kimi

Final Registration Result

Bone position is aligned but orientation is flipped.

16 of 38

Test No. 1

People: Abhishek, Kimi

Moving forward in our surgical procedure, so for M.F.8. Here, the system takes a proactive role, expertly drilling holes in the bone. But it's not about drilling just any hole. Our commitment, as emphasized in M.P.8.1, is to an unparalleled precision. We strive to drill with a positional accuracy of just 2 mm and an orientation fineness within a mere 1.5°. This ensures that every drilling action is not only precise but also aligned with the overarching surgical plan.

Visuals play a paramount role in surgical procedures. Otherwise the doctor will act as a fool during the whole procedure,right? That's why M.F.9 is pivotal. Before making any move, our system will present the surgical roadmap, allowing for real-time visualization of the plan. And when we say real-time, we genuinely mean it. M.P.9 ensures that the visualization is rendered with a latency of 150 ms or even less, offering clinicians an almost instantaneous view.

Lastly, to ensure our system is not just smart but also user-friendly, M.F.10 focuses on the interaction. Our User Interface (UI) is designed to respond swiftly to commands. As per M.P.10.1, any command input will be processed in less than 50 milliseconds. Furthermore, M.P.10.2 lays down the essential control functions our UI will facilitate: starting the procedure, pausing it if needed, and resetting for a fresh start. This trio ensures a seamless user experience and operational control at every stage

MF 11 further establish the obstacle avoiding algorithms here, and MF12 make sure we could handle any emergency situation, restricted on the latency time and responding time

17 of 38

Goal: Gazebo/MoveIt setup and KUKA in simulation

People: Shivangi, Li-Wei

18 of 38

Goal: Gazebo/MoveIt setup and KUKA in simulation

People: Shivangi, Li-Wei

19 of 38

Test No. 2

People: Shivangi, Li-Wei

Moving forward in our surgical procedure, so for M.F.8. Here, the system takes a proactive role, expertly drilling holes in the bone. But it's not about drilling just any hole. Our commitment, as emphasized in M.P.8.1, is to an unparalleled precision. We strive to drill with a positional accuracy of just 2 mm and an orientation fineness within a mere 1.5°. This ensures that every drilling action is not only precise but also aligned with the overarching surgical plan.

Visuals play a paramount role in surgical procedures. Otherwise the doctor will act as a fool during the whole procedure,right? That's why M.F.9 is pivotal. Before making any move, our system will present the surgical roadmap, allowing for real-time visualization of the plan. And when we say real-time, we genuinely mean it. M.P.9 ensures that the visualization is rendered with a latency of 150 ms or even less, offering clinicians an almost instantaneous view.

Lastly, to ensure our system is not just smart but also user-friendly, M.F.10 focuses on the interaction. Our User Interface (UI) is designed to respond swiftly to commands. As per M.P.10.1, any command input will be processed in less than 50 milliseconds. Furthermore, M.P.10.2 lays down the essential control functions our UI will facilitate: starting the procedure, pausing it if needed, and resetting for a fresh start. This trio ensures a seamless user experience and operational control at every stage

MF 11 further establish the obstacle avoiding algorithms here, and MF12 make sure we could handle any emergency situation, restricted on the latency time and responding time

20 of 38

Goal: Bone mount setup in flexed position

Figure: Picture of the bone in flexion mounted on the bone mount

People: Kimi, Li-Wei

21 of 38

Goal: Bone mount setup in flexed position

Figure: Picture of the bone in flexion mounted on the bone mount

People: Kimi, Li-Wei

22 of 38

Goal: End-to-end teleoperated drilling of holes into bone

Clip: Video of teleoperated end-to-end drilling

People: Paramjit

23 of 38

Realizations

Drill choice meets the requirements
Bone needs to be mounted and placed properly instead of being held by hand
End effector design meets the requirements

People: Paramjit

Figure: Picture of the hole drilled in the bone model

24 of 38

Test No. 3

People: Paramjit

Moving forward in our surgical procedure, so for M.F.8. Here, the system takes a proactive role, expertly drilling holes in the bone. But it's not about drilling just any hole. Our commitment, as emphasized in M.P.8.1, is to an unparalleled precision. We strive to drill with a positional accuracy of just 2 mm and an orientation fineness within a mere 1.5°. This ensures that every drilling action is not only precise but also aligned with the overarching surgical plan.

Visuals play a paramount role in surgical procedures. Otherwise the doctor will act as a fool during the whole procedure,right? That's why M.F.9 is pivotal. Before making any move, our system will present the surgical roadmap, allowing for real-time visualization of the plan. And when we say real-time, we genuinely mean it. M.P.9 ensures that the visualization is rendered with a latency of 150 ms or even less, offering clinicians an almost instantaneous view.

Lastly, to ensure our system is not just smart but also user-friendly, M.F.10 focuses on the interaction. Our User Interface (UI) is designed to respond swiftly to commands. As per M.P.10.1, any command input will be processed in less than 50 milliseconds. Furthermore, M.P.10.2 lays down the essential control functions our UI will facilitate: starting the procedure, pausing it if needed, and resetting for a fresh start. This trio ensures a seamless user experience and operational control at every stage

MF 11 further establish the obstacle avoiding algorithms here, and MF12 make sure we could handle any emergency situation, restricted on the latency time and responding time

25 of 38

Additional Progress

2

26 of 38

Manipulator Issue

Arm would stop responding if left idle for a while
Worked with our Smith & Nephew point of contact to identify issue.
Identified issue to be expiring break tests
Contacted Kuka support as discussed.
Issue resolved

People: Paramjit, Abhishek

Figure: Picture of the Kuka pendant with the issue

27 of 38

Workbench Issue

People: Paramjit, Li-Wei

Figure: Screenshot of the missing topologies in the official workbench

Figure: Screenshot of the resolved issue in workbench

28 of 38

D435 Camera Mount

People: Li-Wei

Figure: Picture of the 3D printed D435 camera mount

29 of 38

Challenges & Issues

3

30 of 38

Version incompatibility for ROS integration

We need to push a server on the robot controller which can talk to ROS

Our robot runs SunriseOS version 2.5.
Most open source code is written for version 1.15

Figure: Screenshot of the issues log on the server project

Not only did we adjusted the numerical value for each performance requirement to make them closer to all the existing tech, like here the changed MP1,

But also we modified them to better show all the details of our system’s expected performance.

For example, as you can see here, we seperated the mp2 to better illustrate how the workspace capturing process is going on, also we did that to the original MF 4, now the information of our desired coordinates, including the pose and the depth, are set separately.

MF2/MP2:

Feedback: What portion of the workspace does the system have to sense? Is time a consideration?

Portion: Big enough to accommodate the arm and the bones

Time: not a CONSIDERATION

MF4/MP4:

Feedback: What would be a desired drilling pose? Also separate "obtain depth accurately" as a separate requirement.

Drilling pose: a drilling pose would be a 3D coordinates + normal vector on preoperative STL bone model.

Separation: we separate drilling depth as MF5/MP5

31 of 38

Version incompatibility for ROS integration - UPDATE

We got a response for the issue we created, and they are starting to work on the same.

Figure: Screenshot of the comments on the issueslog on the server project

Not only did we adjusted the numerical value for each performance requirement to make them closer to all the existing tech, like here the changed MP1,

But also we modified them to better show all the details of our system’s expected performance.

For example, as you can see here, we seperated the mp2 to better illustrate how the workspace capturing process is going on, also we did that to the original MF 4, now the information of our desired coordinates, including the pose and the depth, are set separately.

MF2/MP2:

Feedback: What portion of the workspace does the system have to sense? Is time a consideration?

Portion: Big enough to accommodate the arm and the bones

Time: not a CONSIDERATION

MF4/MP4:

Feedback: What would be a desired drilling pose? Also separate "obtain depth accurately" as a separate requirement.

Drilling pose: a drilling pose would be a 3D coordinates + normal vector on preoperative STL bone model.

Separation: we separate drilling depth as MF5/MP5

32 of 38

Risk Management

4

33 of 38

Risks Managed

3d printer is not available / damaged

We printed our hardware early

Robot doesn’t arrive on time

We initiated the process last semester itself, have hardware set up

Manipulator sustains damage during transport

We transported the manipulator safely

Drill isn’t sufficiently strong

We compared and chose a drill with a torque that felt sufficient

We tried drilling in the bone model, and validated our choice

Realsense data too inaccurate

We calibrated the cameras, and the data now meets expectations

Not only did we adjusted the numerical value for each performance requirement to make them closer to all the existing tech, like here the changed MP1,

But also we modified them to better show all the details of our system’s expected performance.

For example, as you can see here, we seperated the mp2 to better illustrate how the workspace capturing process is going on, also we did that to the original MF 4, now the information of our desired coordinates, including the pose and the depth, are set separately.

MF2/MP2:

Feedback: What portion of the workspace does the system have to sense? Is time a consideration?

Portion: Big enough to accommodate the arm and the bones

Time: not a CONSIDERATION

MF4/MP4:

Feedback: What would be a desired drilling pose? Also separate "obtain depth accurately" as a separate requirement.

Drilling pose: a drilling pose would be a 3D coordinates + normal vector on preoperative STL bone model.

Separation: we separate drilling depth as MF5/MP5

34 of 38

Risks to look out for

Off-the shelf dremel might not interface with our arm

We chose the dremel with a foot pedal so that it can be interfaced easily

Control to drill is lost and it injures someone

We will incorporate an e-stop in the drill interface PCB

A group member is unwell

We have a primary and secondary person for each task, so that at least one person is available

Not only did we adjusted the numerical value for each performance requirement to make them closer to all the existing tech, like here the changed MP1,

But also we modified them to better show all the details of our system’s expected performance.

For example, as you can see here, we seperated the mp2 to better illustrate how the workspace capturing process is going on, also we did that to the original MF 4, now the information of our desired coordinates, including the pose and the depth, are set separately.

MF2/MP2:

Feedback: What portion of the workspace does the system have to sense? Is time a consideration?

Portion: Big enough to accommodate the arm and the bones

Time: not a CONSIDERATION

MF4/MP4:

Feedback: What would be a desired drilling pose? Also separate "obtain depth accurately" as a separate requirement.

Drilling pose: a drilling pose would be a 3D coordinates + normal vector on preoperative STL bone model.

Separation: we separate drilling depth as MF5/MP5

35 of 38

Plans

5

36 of 38

Future plans

Figure out how to install LBRServer on Control PC so we can programmatically control the robot.
On Perception side we have found out more approaches we can use -

Feature Metric Registration - https://github.com/XiaoshuiHuang/fmr
Probreg - https://github.com/neka-nat/probreg

We intend to try some of these to achieve more robust registration especially with our new bone setup.

Currently we load saved .ply files. We plan to make a ROS2 pipeline to directly stream point cloud data.

Progress on Drill End Effector Design and PCB Design
Update URDF with End Effector + Table.

Not only did we adjusted the numerical value for each performance requirement to make them closer to all the existing tech, like here the changed MP1,

But also we modified them to better show all the details of our system’s expected performance.

For example, as you can see here, we seperated the mp2 to better illustrate how the workspace capturing process is going on, also we did that to the original MF 4, now the information of our desired coordinates, including the pose and the depth, are set separately.

MF2/MP2:

Feedback: What portion of the workspace does the system have to sense? Is time a consideration?

Portion: Big enough to accommodate the arm and the bones

Time: not a CONSIDERATION

MF4/MP4:

Feedback: What would be a desired drilling pose? Also separate "obtain depth accurately" as a separate requirement.

Drilling pose: a drilling pose would be a 3D coordinates + normal vector on preoperative STL bone model.

Separation: we separate drilling depth as MF5/MP5

37 of 38

Thank you and Break a Leg!

38 of 38

Please keep this slide for attribution

CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon and infographics & images by Freepik