1 of 36

In-bed Deployment of Multimodal Visual Learning for Pressure Reconstruction and Pressure Ulcer Prevention

Zoë LaLena

Advisor: Zackory Erickson

1

2 of 36

Pressure Ulcers

Painful and hard to treat
Affect 2.5 million people each year in the US
$26.8 billion each year

2

Let’s first start off with what causes a pressure ulcer and what it is

We are constantly adjusting our positions, moving pressure from some parts of out bodies to other parts. This is because when pressure is applied on the same part of the body for a prolonged period of time, the blood supply to area is interpreted, causing the area to break down. Pressure ulcers, or bed sores, are open wounds, they commonly affect those bed ridden on ears of the body that have frequent contact with a bed. They are incredibly painful and the worse they are the harder they are to treat. Pressure ulcers start at stage 1 affecting the skin and affect further down in to tissue, muscle and eventually bone at stage four.

Pressure injuries acquired in a hospital setting cost an estimate $26.8 billion in the US each year. This is because these types of open wound tend to be difficult to treat increase the risk of infections , and over 2.5 million people are affected by pressure ulcers in the us each year.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7948545/

https://www.nhsinform.scot/illnesses-and-conditions/skin-hair-and-nails/pressure-ulcers/#:~:text=Pressure%20ulcers%20are%20caused%20by,to%20help%20keep%20tissue%20healthy.

https://www.ahrq.gov/topics/pressure-ulcers.html#:~:text=Each%20year%2C%20more%20than%202.5,United%20States%20develop%20pressure%20ulcers.

3 of 36

BodyMap

Reference

Depth

Pressure

Reference

Body Mesh - Pressure

3

4 of 36

Beyond Pressure Ulcers

Sleep related motion disorders
Sleep quality assessment
Epilepsy monitoring
ICUs
Robotic assistance

Puthuveetil, K., Kemp, C. C., & Erickson, Z. (2022). Bodies uncovered: Learning to manipulate real blankets around people via physics simulations

4

5 of 36

BodyMAP

Depth Image

Pressure Image

3D Pressure

Distribution

(Front & Back)

RGB

Without

Cover

Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed

Featu Indexing Module

ResNet

SMPL

PointNet

BodyMAP

references

inputs

outputs

5

The bodymap model takes both depth image and pressure image and output a 3d human mesh with the distribution of pressure value. The red region means more pressure.

It first passes through a Resnet to extract feature from original images and using smpl model to generate a human mesh.

Then it uses a Feature index module to fuse mesh with other features, and finally use a pointnet to get pressure map over the human body mesh.

Let's start at the end and take a look at what we are expecting to get out from this model, a 3D pose and shape of the patient and the corresponding 3d pressure map. As you can see here we have a 3d model and the brighter/redder the more pressure. This model takes in 3 inputs, a depth image, acquired from a camera above the bed, a pressure map from a pressure mat underneath the patient and the patient's gender (this lets us know what smpl model to use, male or female). RGB is not an input into the model but for visualization purposes, you can see the rgb image corresponding to this data from the SLP dataset or Simultaneously-Collected Multimodal Lying Pose dataset collected by Lui et. al. This test data also contains the same pose without a covering.

6 of 36

BodyMAP

Jointly Predicting Body Mesh and 3D Applied Pressure Map for People in Bed

6

7 of 36

Deployment

7

8 of 36

Deployment

Capture Data

Run BodyMap

Display Output

8

9 of 36

Motivation

Depth

Pressure

Reference

Body Mesh - Pressure

9

10 of 36

Why Does This Happen?

Dynamic environments
Diverse bodies
Differing setups

https://www.shutterstock.com/video/clip-4820933-caring-medical-staff-moving-patient-hospital-bed

10

11 of 36

Differing Distributions

Training

Real World

Transformed Real World

Some Transform

11

12 of 36

Deployment

Capture Data

Run BodyMap

Display Output

12

13 of 36

Deployment

Capture Data

Transform Data

Run BodyMap

Display Output

13

14 of 36

Capture Data

Mountable Surface

RealSense RBG & Depth Camera

Moveable Hospital Bed

Pressure Sensing Mat

ArUco Markers

14

15 of 36

Collected Data

RGB

Depth

Pressure

15

16 of 36

Deployment – Transformation

16

17 of 36

SMPL

17

18 of 36

What is SMPL?

Realistic 3D model of the human body
Based on skinning and blend shapes
learned from thousands of 3D body scans

https://smpl-x.is.tue.mpg.de/

Input

Major

Joints

Skeleton

SMPL

SMPL-X

18

19 of 36

SMPL-X

4 years after SMPL
Improved error
Facial features and hand/finger positions

Model	Keypoints	V2V Error	Joint Error
SMPL	Body	57.6	63.5
SMPL-X	Body+Hands+Face	52.9	62.6

Pavlakos, Georgios, et al. "Expressive body capture: 3d hands, face, and body from a single image." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.

19

20 of 36

Getting “Ground Truth” SMPL Models

SMPLX to SMPL

SMPLX

Openpose

20

21 of 36

Errors

21

22 of 36

Error & Differences: V2V

Mean squared Error between over 6000 vertex positions on the meshes

The same vertices should have similar positions

22

23 of 36

Error & Differences: Joint Error

Zeng, Qiang, Gang Zheng, and Qian Liu. "DTP: learning to estimate full-body pose in real-time from sparse VR sensor measurements." Virtual Reality 28.2 (2024): 1-17.

Mean squared Error between 45 joint positions

23

24 of 36

Transforming

24

25 of 36

Transforming

Histogram Matching

CycleGan

CVAE

Transformer

https://towardsdatascience.com/a-gentle-introduction-to-cycle-consistent-adversarial-networks-6731c8424a87

25

26 of 36

Histogram Matching

26

27 of 36

Histogram Matching: Results

27

28 of 36

CycleGan

Generator A2B

Discriminator A

Decision [0,1]

Discriminator B

Decision [0,1]

Generator B2A

Input A

Cyclic A

Generated B

https://towardsdatascience.com/a-gentle-introduction-to-cycle-consistent-adversarial-networks-6731c8424a87

28

29 of 36

Cycle-Gan Results: Depth

Real World Image

Training Set Image

Generated Image

(real world to training set)

Generated Image

(training set to real world)

Generated Image

(training set to real world)

Generated Image

(real world to training set)

Real World Pressure Image

Training Set Pressure Image

Generated Image

(real world to training set)

Generated Image

(training set to real world)

Generated Image

(training set to real world)

Generated Image

(real world to training set)

29

30 of 36

Cycle-Gan Results: Pressure

Real World Pressure Image

Training Set Pressure Image

Generated Image

(real world to training set)

Generated Image

(training set to real world)

Generated Image

(training set to real world)

Generated Image

(real world to training set)

Real World Image

Training Set Image

Generated Image

(real world to training set)

Generated Image

(training set to real world)

Generated Image

(training set to real world)

Generated Image

(real world to training set)

30

31 of 36

CVAE: Conditional Variational Auto-Encoders

c

Encoder

Decoder

c

…

Latent Distribution

Mean

Standard Deviation

Sampled Latent Vector

Covariate Information

Abdelli, Khouloud, et al. "Degradation prediction of semiconductor lasers using conditional variational autoencoder." Journal of Lightwave Technology 40.18 (2022): 6213-6221.

31

32 of 36

Transformers

ITTR: Unpaired Image-to-Image Translation with Transformers

32

33 of 36

Transformers

InstaFormer: Instance-Aware Image-to-Image Translation with Transformer

33

34 of 36

Summary

Workflow to get data and run BodyMap

Collected nearly 200 RGB, pressure, and depth images

Workflow to get “ground truth” data from RGB images

Ground truth allows us to see how well we are doing, calculate error

Working on data transformation

34

35 of 36

Thank you

35

36 of 36

Works Cited

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7948545/
https://www.nhsinform.scot/illnesses-and-conditions/skin-hair-and-nails/pressure-ulcers/#:~:text=Pressure%20ulcers%20are%20caused%20by,to%20help%20keep%20tissue%20healthy.
https://www.ahrq.gov/topics/pressure-ulcers.html#:~:text=Each%20year%2C%20more%20than%202.5,United%20States%20develop%20pressure%20ulcers.
BodyMAP paper
Simultaneously-Collected Multimodal Lying Pose Dataset: Towards In-Bed Human Pose Monitoring under Adverse Vision Conditions

36