1 of 80

DIGITAL HEALTHCARE USING CONSUMER DEVICES AND CLOUD COMPUTING

Peter Yiğitcan Washington

Assistant Professor

Information & Computer Sciences

University of Hawaiʻi at Mānoa

2 of 80

Opportunities in Consumer Digital Health

1

3 of 80

Cloud Computing Situations Discussed

AI and Privacy - Training Artificial Intelligence (AI) using Protected Health Information (PHI)

Crowd Computing - Cloud-based crowdsourcing platforms like Amazon Mechanical Turk

Federated Learning - Computational offloading of device-specific AI models to the cloud

4 of 80

Opportunities in Consumer Digital Health

1

5 of 80

Opportunities in Consumer Digital Health

1

6 of 80

Opportunities in Consumer Digital Health

1

7 of 80

Opportunities in Consumer Digital Health

1

8 of 80

Opportunities in Consumer Digital Health

1

9 of 80

Opportunities in Consumer Digital Health

1

10 of 80

Autism

Affects 1 in 40 children in the United States¹

Lack of eye contact, trouble with social responsiveness, difficulty with emotion recognition and evocation, distinguishable vocal prosody, repetitive body motions, self harm, …

Image: Raising Children Network

1. Kogan et al. Pediatrics. 2018.

2

11 of 80

“Autism is a Spectrum”

3

12 of 80

“Autism is a spectrum” �“Autism Consists of Multiple Spectrums”

Kim et al., Journal of the American Academy of Child & Adolescent Psychiatry. 2018.

DeBoth et al., Research in Autism Spectrum Disorders. 2017.

Rapin et al., Developmental Neuropsychology. 2009.

Gabis et al., Journal of Child Neurology. 2008.

Hrdlicka et al., European Child and Adolescent Psychiatry. 2005.

Tager-Flusberg et al., Philosophical Transactions of the Royal Society of London. 2003.

Eaves et al., Journal of Autism and Developmental Disorders. 1994.

Volkmar et al., Journal of the American Academy of Child & Adolescent Psychiatry. 1989.

4

13 of 80

Diagnostic Bottlenecks

Long questionnaires to diagnose, but only a subset required^1,2

Extensive wait times for current standard of care: 18+ months³

Systemic discrimination of underserved and marginalized populations^4,5

Wall et al. Translational Psychiatry. 2012.
Washington et al. Pacific Symposium on Biocomputing. 2020.
Gordon-Lipkin et al. Pediatric Clinics. 2016.
Nazroo et al. American Journal of Public Health. 2003.
Ning, …, Washington et al. Journal of Medical Internet Research. 2019.

5

14 of 80

Digital Diagnostics

6

15 of 80

The Diagnostic Process in Psychiatry

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

7

16 of 80

The Diagnostic Process in Psychiatry

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

2

7

17 of 80

The Diagnostic Process in Psychiatry

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

2

0

2

3

0

7

18 of 80

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

2

0

2

3

0

ASD

The Diagnostic Process in Psychiatry

7

19 of 80

The Diagnostic Process in Psychiatry

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

B

C

…

2

3

0

3

2

0

3

1

3

0

1

3

ASD

NT

7

20 of 80

The Diagnostic Process in Psychiatry

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

B

C

…

2

3

0

3

2

0

3

1

3

0

1

3

ASD

NT

X

y

7

21 of 80

Which Features Should we Prioritize?

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

B

C

…

2

3

0

3

2

0

3

1

3

0

1

3

ASD

NT

Not related

to ASD

Wall et al. Translational Psychiatry. 2012.

Tariq, …, Washington et al. PLoS Medicine. 2018.

Washington et al. Pacific Symposium on Biocomputing. 2019.

7

22 of 80

Patient ID

Emotion

Expression

Eating

Frequency

Prosody

Hand

Mannerisms

Eye Contact

Diagnosis

A

B

C

…

2

3

0

3

2

0

3

1

3

0

1

3

ASD

NT

Not related

to ASD

Very related

to ASD

Wall et al. Translational Psychiatry. 2012.

Tariq, …, Washington et al. PLoS Medicine. 2018.

Washington et al. Pacific Symposium on Biocomputing. 2019.

7

Which Features Should we Prioritize?

23 of 80

Computer Vision Algorithms Cannot Predict Complex Behaviors (yet)

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

8

24 of 80

Humans Are Good At This!

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

9

25 of 80

Humans Are Good At This!

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

9

26 of 80

Crowdsourcing Platforms

27 of 80

Crowdsourcing Experiment Interface

Washington et al. Scientific Reports. 2021.

10

28 of 80

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

Evaluating Workers

11

29 of 80

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

Evaluating Workers

11

30 of 80

Does the child enjoy participating in social games and interactions?

Always

Never

Sometimes

1

Evaluating Workers

11

31 of 80

Does the child entertain herself/himself/themself without help from others?

Always

Never

Sometimes

1

2

Evaluating Workers

11

32 of 80

Rate the child’s expressive language and conversation ability.

Excellent

Poor

Satisfactory

1

2

0

Evaluating Workers

11

33 of 80

Does the child understand and respond to spoken language?

Always

Never

Sometimes

1

2

0

1

Evaluating Workers

11

34 of 80

Would the child comfort someone if they were feeling upset, hurt, or sad?

Yes

Unlikely

Probably

1

2

0

1

0

Evaluating Workers

11

35 of 80

1

2

0

1

0

Evaluating Workers

12

36 of 80

Gold

Standard Classifier

1

2

0

1

0

Evaluating Workers

f(X)

12

37 of 80

Gold

Standard Classifier

89.3%

Autism

1

2

0

1

0

Evaluating Workers

f(X)

12

38 of 80

Gold Standard Classifier Training

Patient ID

Emotion

Expression

Social

Interactions

Eye Contact

Diagnosis

A

B

C

2

3

0

3

1

3

0

1

3

ASD

NT

…

f(X)

y

13

39 of 80

89.3%

Autism

12.4%

Autism

70.9%

Autism

3.3%

Autism

99.0%

Autism

96.7%

Autism

8.3%

Autism

85.2%

Autism

89.3%

Autism

22.1%

Autism

Evaluating Humans in the Loop

14

40 of 80

89.3%

Autism

12.4%

Autism

70.9%

Autism

3.3%

Autism

99.0%

Autism

96.7%

Autism

8.3%

Autism

85.2%

Autism

89.3%

Autism

22.1%

Autism

Evaluating Humans in the Loop

14

41 of 80

8 out of 8

7 out of 8

6 out of 8

5 out of 8

4 out of 8

3 out of 8

2 out of 8

0 out of 8

Evaluating Humans in the Loop

15

42 of 80

8 out of 8

7 out of 8

6 out of 8

5 out of 8

4 out of 8

3 out of 8

2 out of 8

0 out of 8

Washington et al. Pacific Symposium on Biocomputing. 2021.

Washington et al. Journal of Personalized Medicine. 2020.

Washington et al. Journal of Medical Internet Research. 2019.

Evaluating Humans in the Loop

15

43 of 80

Performance of Top Workers

Washington et al. Scientific Reports. 2021.

16

44 of 80

Diagnostic Difficulty

Washington et al. Scientific Reports. 2021.

17

45 of 80

Patient Privacy

Washington et al. Scientific Reports. 2021.

18

46 of 80

Patient Privacy

Washington et al. Scientific Reports. 2021.

19

47 of 80

Patient Privacy

Washington et al. Scientific Reports. 2021.

LR5: 88% accuracy, 96% sensitivity, 80% specificity

19

48 of 80

Patient Privacy

Washington et al. Scientific Reports. 2021.

LR5: 88% accuracy, 96% sensitivity, 80% specificity

LR10: 80% accuracy, 100% sensitivity, 60% specificity

19

49 of 80

Gaussian Blurring

Washington et al. Intelligence Based Medicine. 2022.

20

50 of 80

Pixelation

Washington et al. Intelligence Based Medicine. 2022.

21

51 of 80

Optical Flow

Washington et al. Intelligence Based Medicine. 2022.

22

52 of 80

Cloud Computing Implications

PHI data must be stored on cloud platforms that are HIPAA-compliant

Humans in the loop can be on cloud platforms like MTurk for non-protected health information (non-PHI)

Humans in the loop for PHI must be on a HIPAA-compliant annotation platform (e.g., custom built)

53 of 80

HIPAA Compliant Options

AWS

Google Cloud

Microsoft Azure

…

54 of 80

Hosting a HIPAA-Compliant Crowdsourcing Interface on the Cloud

S3 for storage

EC2/EBS for web hosting

DynamoDB for database

Washington et al. JMIR. 2022.

55 of 80

Best Practices for Ensuring HIPAA Compliance on AWS

Disable public access
Define logical boundaries
Schedule patching
Separate patient information from orchestration
Enable encryption
Use IAM roles instead of the root account
Implement person or entity authentication
Enable data backup and disaster recovery
Use audit logging and monitoring controls
Use encryption features native to HIPAA-eligible services
Ensure technical controls and security safeguards are in place for each cloud resource
Use encrypted data volumes for EC2 instances
Don't make S3 buckets open to the public
Enable backup for RDS instances

56 of 80

Opportunities in Consumer Digital Health

57 of 80

Opportunities in Consumer Digital Health

58 of 80

Personalized Machine Learning

AI Model 1

AI Model 2

AI Model 3

24

59 of 80

25

60 of 80

Detecting Substance Craving and Use

26

61 of 80

Detecting Stress and Stress-Induced BP Spikes

27

62 of 80

The Key: Personalized �Self-Supervised Learning (SSL)

28

63 of 80

What is Self-Supervised Learning (SSL)?

29

64 of 80

SSL with Wearable Biosignals

30

65 of 80

SSL with Wearable Biosignals

31

66 of 80

Benefits of Personalization without SSL

Model Type	Accuracy (mean +/- standard deviation)	F1-Score (mean +/- standard deviation)
Personalized	95.06 +/- 9.24	91.72 +/- 15.33
Subject-Inclusive Generalized	66.95 +/- 13.76	42.50 +/- 17.37
Subject-Exclusive Generalized	67.65 +/- 13.48	43.05 +/- 17.20

Task: stress prediction using wearable sensors

Li, Washington. Journal of Medical Internet Research AI. 2024. (under revision)

32

67 of 80

Benefits of SSL without Personalization

Task: emotion prediction using audio

Nimitsurachat, Washington. AI. 2024.

33

68 of 80

Benefits of Personalized SSL

Task: stress prediction using wearable sensors

Islam, Washington. International Conference on Machine Learning (ICML) AI+HCI Workshop. 2023.

34

69 of 80

Challenges with Personalization

Task: nurse stress prediction using wearable sensors

Eom, …, Washington. International Conference on Machine Learning (ICML) Multimodal Healthcare Workshop. 2023.

35

70 of 80

Challenges with Personalization

Task: nurse stress prediction using wearable sensors

Eom, …, Washington. International Conference on Machine Learning (ICML) Multimodal Healthcare Workshop. 2023.

Inconsistent labeling across nurses (between subjects)

Inconsistent labeling on different days (within subjects)

Noisy data

…

36

71 of 80

Human-Computer Interaction Innovations

37

72 of 80

Human-Computer Interaction Innovations

38

73 of 80

Cloud Computing Implications

Opportunities for federated learning approaches

Cloud needs to be engaged for each user

Cloud needs to be repeatedly engaged per user

74 of 80

Where Does the Training and Pre-training Occur?

Pre-training:

Cloud pre-training
On-device pre-training

Fine-tuning:

Cloud-fine tuning
On-device fine-tuning

75 of 80

Decision Making Process for Training

76 of 80

Federated Learning

77 of 80

Opportunities in Consumer Digital Health

78 of 80

Cloud Computing Situations Discussed

AI and Privacy - Training Artificial Intelligence (AI) using Protected Health Information (PHI)

Crowd Computing - Cloud-based crowdsourcing platforms like Amazon Mechanical Turk

Federated Learning - Computational offloading of device-specific AI models to the cloud

79 of 80

Acknowledgements

Mentors: Dennis P. Wall, PhD; Nick Haber, PhD; Terry Winograd, PhD

Research Collaborators: Dennis P. Wall, PhD (Stanford); Nick Haber, PhD (Stanford); Kristina Phillips, PhD (Kaiser Permanente); Finale Doshi-Velez, PhD (Harvard); Roberto Benzo, PhD (Ohio State); Jennette Moreno, PhD (Baylor College of Medicine); Chanhyun Park, PhD (UT Austin); Pallav Pokhrel, PhD (Hawaii); Alexander Stokes, PhD (Hawaii); John Shepherd, PhD (Hawaii)

Clinical Expertise: Dr. Anthony Guerrero, MD (Hawaii); Dr. Gerald Busch, MD (Hawaii); Dr. Carl Feinstein, MD (retired); Dr. Gary Darmstadt, MD (Stanford); Dr. Sally Ozonoff, PhD (UC Davis); Dr. Meghan Miller, PhD (UC Davis); Dr. Anne Arnett, PhD (Harvard / Boston Children’s)

Trainees: Aditi Jaiswal, Lydia Sollis, Zerin Tumpa, Ali Kargarandehkordi, Chris Slade, Peranut Nimitsurachat, Kaiying Lin, Johannes Fuest, Yang Qian, Tanvir Islam, Zain Jabbar, Yinan Sun, Armin Soltan, Ryan Fitzpatrick, Amanda Nitta, Rahat Zawad

Undergraduate and High School Students: Sunny Eom, Annie Eom, Shubham Parab, Joe Li, Jing Zheng, Tishya Chhabra, Cathy Hou, Nathan Chi, Anish Lakkapragada, Agnik Banerjee, Essam Sleiman, Ritik Patnaik

Funding: PI: NIH New Innovator Award (DP2EB035858); PI: AIM-Ahead (OT2OD032581); PI: Center for Pacific Innovations, Knowledge, and Opportunities (U54GM138062); PI: Ola Hawaii (U54MD007601); PI: Hawaii Community Foundation; PI: Amazon Web Services Cloud Credit for Research; Co-I: NIH R01HD112349; Co-I: AIM-Ahead (OT2OD032581)

40

80 of 80

IMPROVING CONSUMER DIGITAL HEALTH INFORMATICS

Peter Yiğitcan Washington

Assistant Professor

Information & Computer Sciences

University of Hawaiʻi at Mānoa