1 of 129

Transforming Health Care Through Artificial Intelligence Revolutions

Ram D. Sriram, Ph.D.

Chief, Software and Systems Division/ITL

National Institute of Standards and Technology

URL: http://www.nist.gov/itl/ssd/

sriram@nist.gov

2022-04-19

1

Transformation of Health Care Through AI Revolutions

2 of 129

Outline

AI: A Brief History
NIST and AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

2

Transformation of Health Care Through AI Revolutions

3 of 129

What is Intelligence?�(Dictionary Definitions)

The ability to meet (novel) situations successfully by proper behavior adjustments

The ability to perceive the interrelationships of presented facts in such a way as to guide action toward a desired goal.

3

Transformation of Health Care Through AI Revolutions

4 of 129

Intelligent Agent: Attributes

Have mental attitudes
Perceive and model external worlds
Understand
Solve problems (innovativeness)
Generalize/learn
Plan and predict
Use language
Know limits

4

Transformation of Health Care Through AI Revolutions

5 of 129

Intelligent Agent: What About?

Awareness/Consciousness
Aesthetic appreciation
Emotion
Sensory acuteness
Muscular coordination

See Frames of Mind

5

Transformation of Health Care Through AI Revolutions

6 of 129

Artificial Intelligence: A Simple View

6

Artificial intelligence is concerned with the development of

computer programs that emulate the intelligence of humans.

Elaine Rich

ACTION

MANIPULATORS

PERCEPTION

SENSORS

ENVIRONMENT

INTELLIGENCE AGENT

Transformation of Health Care Through AI Revolutions

7 of 129

Evolution of AI

Prehistory (Paleolithic: Indian and Greek Philosophies; Neolithic: Hume, Russell, Turing)
The Beginning (1956: AI coined at Dartmouth Conference)
Early years and Rise of Knowledge Systems (1957 – 1980: GPS, Dendral, Hearsay, Macsyma, Mycin, Shrdlu, Shakey, etc.)
Knowledge is Power & Early Neural Networks (1980 -- 1990: The first wave)
The Silent Period (1990 – 2000: Deep Blue, Rise of Robots, Commercialization of AI technologies, e.g., Speech Recognition)
Neural Networks to the Fore (2000 – 2015: The second wave: Deep Learning)
Symbiosis of Neural and Knowledge Networks (2015 – 2025: Explainable AI, OKN, etc. – The third wave)
The Conscious Machine (2025 -- ?? -- The fourth wave: Tsunami)

(see also Henry Kautz’s talk: https://www.cs.rochester.edu/u/kautz/talks/index.html and associated paper: https://onlinelibrary.wiley.com/doi/full/10.1002/aaai.12036)

7

Transformation of Health Care Through AI Revolutions

8 of 129

Outline

AI: A Brief History
NIST and AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs + NNs
Future & Summary

8

Transformation of Health Care Through AI Revolutions

9 of 129

NIST’s Mission

To promote U.S. �innovation and industrial �competitiveness by �advancing measurement �science, standards, �and technology in ways that enhance economic security �and improve our quality of life.

9

Transformation of Health Care Through AI Revolutions

10 of 129

(AI for NIST, NIST for AI)

10

TRUSTWORTHY AI

MEASURES & STANDARDS

Research

Investigation

Application

Deployment

Courtesy: Mike Garris

Transformation of Health Care Through AI Revolutions

11 of 129

NIST for AI

11

TRUSTWORTHY AI

MEASURES & STANDARDS

Privacy

Security

Explainabe

Resilience

Reliability

Research

Investigation

Application

Deployment

Courtesy: Mike Garris

Transformation of Health Care Through AI Revolutions

12 of 129

AI for NIST

12

TRUSTWORTHY AI

MEASURES & STANDARDS

IoT

Robotics

Material Science

Smart Manufacturing

Biomedical Imaging

…

Research

Investigation

Application

Deployment

Courtesy: Mike Garris

Transformation of Health Care Through AI Revolutions

13 of 129

NIST AI Program

13

CONDUCT FOUNDATIONAL RESEARCH TO ADVANCE TRUSTWORTHY AI TECHNOLOGIES

ADVANCE AI RESEARCH AND INNOVATION ACROSS THE NIST LABORATORY PROGRAMS

PARTICIPATE AND LEAD IN THE DEVELOPMENT OF STANDARDS TO ADVANCE AI INNOVATION

13

CONTRIBUTE NIST’S TECHNICAL EXPERTISE TO DISCUSSIONS AND DEVELOPMENT OF POLICIES

ENSURE THAT NIST HAS RESOURCES AND EXPERTISE TO CARRY OUT ITS AI PROGRAMS

ESTABLISH BENCHMARKS AND DEVELOP METRICS TO EVALUATE AI TECHNOLOGIES

Courtesy: Reva Schwartz & Elham Tabassi

Transformation of Health Care Through AI Revolutions

14 of 129

Key NIST roles for the Federal Government

14

NIST AI RISK MANAGEMENT FRAMEWORK

HR116-445, NDAA FY21 SEC. 5301

NATIONAL AI ADVISORY COMMITTEE

NDAA FY21 SEC. 5104

FEDERAL AI STANDARDS COORDINATOR

14

INTERAGENCY COORDINATION

WH OSTP/NSTC, TTC, QUAD

STAKEHOLDER OUTREACH

NDAA FY21 SEC. 5302

AI RESEARCH RESOURCE TASK FORCE

NDAA FY21 SEC. 5106

Courtesy: Reva Schwartz & Elham Tabassi

Transformation of Health Care Through AI Revolutions

15 of 129

Past NIST Efforts in Evaluation

15

Transformation of Health Care Through AI Revolutions

16 of 129

Interacting with NIST

Guest Researchers/Faculty Associates
Grants and Contracts
IPA
Summer Students
NRC Post Doctoral Program
Collaborative Proposals

16

Transformation of Health Care Through AI Revolutions

17 of 129

Outline

AI: A Brief History
NIST and AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

17

Transformation of Health Care Through AI Revolutions

18 of 129

Health Care

Facts
Vision
P9 Medicine
Health Care Infrastructure
Electronic Health Records
Toward AI-based Health Care

18

Transformation of Health Care Through AI Revolutions

19 of 129

Healthcare Facts

$4.1 Trillion dollars spent in 2020 on healthcare in the U.S. (http://www.cms.gov)
It is estimated that approximately $750billion is lost due to inefficiencies in the system

Effective use of IT may help reduce these costs

Multiple parties playing different roles

19

Transformation of Health Care Through AI Revolutions

20 of 129

Levels of Biological Information

Ecologies

Societies/Populations

Individuals

Organs

Tissues

Cells

Protein and gene networks

Protein interaction networks

Protein

mRNA

DNA

20

HEALTH CARE

Courtesy: Leroy Hood

BIOSCIENCES

Transformation of Health Care Through AI Revolutions

21 of 129

21

“Genomics”

“Transcriptomics”

“Proteomics”

“Metabolomics”

“Cellomics”

From Genomics to Phenotypes

Lucasio

Courtesy: Laurie Locascio

Transformation of Health Care Through AI Revolutions

22 of 129

22

2D gel

electrophoresis

1975

1990

ESI & MALDI

1980

Gene cloning

Sequencing

1994

Complex mixture analysis

LC-MS(/MS)

Chip-based

approaches

1996

1944

DNA

Genetic

material

1953

DNA

structure

1980

1990

2000

1960

1970

Recombinant

technology

Automated

sequencing

High-throughput at Genome scale,

‘data rich’ biology

1995

Haemophilus

influenzae

first genome sequenced

2001

Human genome

sequenced

1931

Non-equilibrium

thermodynamics

1952

Self-

organization

1957

1980

1990

1970

Feedback regulation

In metabolism

Analog simulation,

bioenergetics, lac operon

Large-scale simulators

of metabolic dissipative

structures, energy coupling MAC and BST

2000

‘data poor’ in silico biology,

models of viruses, red blood cell

Genome-scale models and analysis,

large-scale kinetic models

Proteomics

Genomics

BioAnalysis

Computer

Science

Systems

Biology

(Medicine)

1936

Turing machines

1956

Artificial

Intelligence

1960

Programming

Languages

1980

Commercial

Database

1974

Internet

Bioinformatics

mid 1980

1990

Major advances in AI

2000

Engineering

1943

Finite element

analysis

1950

Systems

engineering

1960

Cybernetics

Integrated

Circuits

1970

1980

Computational

FEM

Mainframe

Computers

1990

VLSI CAD/CAD/

Geometry

Large scale product

Process simulation

PDM/Knowl.

based

Engineering

2000

MEMS

Nano

PLM

PC’S

Supercomputers

Multi-scale Domain

Simulation

Bernhard O. Palsson

Transformation of Health Care Through AI Revolutions

23 of 129

Advances Making Future Health Vision Attainable

23

Future

Health

Vision

Courtesy: Jack Corley

mPCD

Networking

Communications

and Imaging

Advances in

Computing,

Imaging, and

Information

Technology

Speed and

Storage

Software (Internet,

Cloud,

Data Analytics, Etc.)

Advances in Healthcare

Technology

Human Genome

Project

Pharmaceuticals

and

Nutraceuticals

Medical

Devices

Evidence-Based

Healthcare

Continuum

of Care

Advances

in

Healthcare

Practice

Disease

Management

Transformation of Health Care Through AI Revolutions

24 of 129

The P9 Concept

Personalized
Predictive
Participatory
Precise (recommendation, decision analytics)
Preventive
Pervasive (including point of care)
Privacy-preserving
Protective (security)
Priced reasonably

24

Based on discussions with Leroy Hood and Ramesh Jain

Transformation of Health Care Through AI Revolutions

25 of 129

25

Health IT Network

Ward

Administration

Radiology

Laboratory

Hospital

Pharmacy

EHR

Health Info Exchanges

Homecare devices

Personal Health Record

Transformation of Health Care Through AI Revolutions

26 of 129

26

From: Procuring Interoperability, National Academy of Medicine

Transformation of Health Care Through AI Revolutions

27 of 129

EHRs: Key Issues

27

Input (user interfaces)

Store (representation and persistency)

Manipulate (search, mining, knowledge creation)

Exchange (syntactic and semantic interoperability)

SECURITY

Transformation of Health Care Through AI Revolutions

28 of 129

Dimension 1: Derivation-Formation Spectrum

Derivation-type Problems [Hypothesize & Test]

Diagnosis
Classification/Interpretation
Control

Formation-Type Problems [Generate & Test]

Planning
Scheduling
Design/Configuration

28

Transformation of Health Care Through AI Revolutions

29 of 129

29

From Eric Topol’s Paper

Dimension 2: Life Cycle

Transformation of Health Care Through AI Revolutions

30 of 129

Outline

AI: A Brief History
NIST And AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

30

Transformation of Health Care Through AI Revolutions

31 of 129

KBeS: A Model

31

Context

(Short Term Memory)

Knowledge-base

(Long Term Memory)

Inference Mechanism

(Thinking Process)

Backward Chaining

Forward Chaining

Constraint Propagation

Hierarchical Generate-Test

Etc.

Rules

Frames

Logic

Transformation of Health Care Through AI Revolutions

32 of 129

32

EXPERT

KNOWLEDGE

BASE

KNOWLEDGE

ACQUISITION FACILITY

INFERENCE

MECHANISM

EXPLANATION

FACILITY

CONTEXT

USER

USER INTERFACE

Schematic of a KBES with Explanation and

Knowledge Acquisition Modules

From: Intelligent Systems for Engineering, Ram D. Sriram

Transformation of Health Care Through AI Revolutions

33 of 129

An Early History of AIM

CASNET/EXPERT (Rutgers University, 1971-1978, Causal Networks, Glaucoma)
MYCIN/EMYCIN (Stanford University,1975, Rule-based, Infectious Diseases)
ANNA and others (MIT, Mid70s, Concepts&Rules, several domains)
INTERNIST/CADUCEUS (Univ. Pittsburgh, mid70s, Causal Networks&Ontologies, General Medicine)

33

Note: A commercial version, based on some of above concepts, called ILLIAD was marketed in the late 1990s

(see Warner, H., Sorenson, D., and Bouhaddou, O., Knowledge Engineering in Health Informatics,

Springer Verlag, 1997 )

Transformation of Health Care Through AI Revolutions

34 of 129

MYCIN: Pioneer in KBeS

34

Transformation of Health Care Through AI Revolutions

35 of 129

Rules in Mycin

35

Transformation of Health Care Through AI Revolutions

36 of 129

36

Explaining Reasoning in MYCIN: During Consultation

Courtesy: Scott, Clancey. Davis, Shortliffe. in Rule-based Expert Systems; http://www.shortliffe.net/

Transformation of Health Care Through AI Revolutions

37 of 129

37

Explaining Reasoning in MYCIN: Conclusion

Courtesy: Scott, Clancey. Davis, Shortliffe. in Rule-based Expert Systems; http://www.shortliffe.net/

Transformation of Health Care Through AI Revolutions

38 of 129

Outline

AI: A Brief History
NIST And AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

38

Transformation of Health Care Through AI Revolutions

39 of 129

39

AI and Machine Learning

Artificial Intelligence

Machine Learning

Neural Networks

Deep learning

Knowledge-based Systems

Transformation of Health Care Through AI Revolutions

40 of 129

40

Models built on extracted features

Models built on raw data

Some Machine Learning Models

Regression models

Decision-tree models

Mixture models

Multilayer perceptron models

Convolutional Neural Network models

Courtesy: Sarala Padi

Transformation of Health Care Through AI Revolutions

41 of 129

Medical Imaging at NIST

41

Develop test methods to evaluate various imaging algorithms.

Develop/validate standard methods and technologies for combining, interpreting, visualizing, and storing/accessing data from various imaging techniques, including MRI, CR-scans, PET, molecular imaging, ultrasound, optical, and other commonly employed clinical methods, to increase their diagnostic power.

Define improved methods for acquiring and displaying images for telemedicine applications

Transformation of Health Care Through AI Revolutions

42 of 129

Medical Imaging

Change Analysis – Lung Cancer
NIST/QIBA Activities
Iterative Reconstruction
Interpreting Wireless Capsule Endoscopy Images
From Images to Diagnosis through Ontologies
Image Quality for Healthcare Applications
Computational Metrology for Biomedical Imaging
Performance of Scalable Systems

42

Transformation of Health Care Through AI Revolutions

43 of 129

Gastroenteroscopy, Colonoscopy vs. WCE

43

Transformation of Health Care Through AI Revolutions

44 of 129

44

Animation: Courtesy of GivenImaging

Transformation of Health Care Through AI Revolutions

45 of 129

Some Technical Info on WCE

WCE system:

Capsule:

1. Optical dome

2. Lens holder

3. Lens

4. Illuminating LEDs

5. CMOS image sensor

6. Battery

7. Radio transmitter

8. Antenna

Receiver
Computer workstation �with software

45

Images: courtesy of GivenImaging

Height: 11mm, Width: 26mm

Weight: 3.7gr

Transformation of Health Care Through AI Revolutions

46 of 129

46

problem

normal

Image segmentation

Feature Vectors Computation

Mapping calculated feature

vectors into disease ontology

Highlighting regions with

deviation from normal conditions

Input Image

Output:

Suggestion of the potential diagnosis

Sugessted diagnosis: Ulcer...

Methodology for Image to Diagnosis Through Disease Ontology

Courtesy: Sub, Mala Ramaiah, DN Reddy, Marcin Kociolek

Transformation of Health Care Through AI Revolutions

47 of 129

Map Feature Vector to Diagnosis

Feature vectors are used to query disease ontology.
Lesion features are mapped to the appropriate disease (Ulcer in our example).

47

Transformation of Health Care Through AI Revolutions

48 of 129

48

Fragment of Disease Ontology in Protege’-Ontoviz

Transformation of Health Care Through AI Revolutions

49 of 129

UML Representation of �Inflammatory Bowel Disease

49

+

e

n

d

o

s

c

o

p

i

c

f

e

a

t

u

r

e

1

:

<

u

n

s

p

e

c

i

f

i

e

d

>

=

u

l

c

e

r

s

+

l

o

c

a

t

i

o

n

:

s

t

r

i

n

g

(

i

d

l

)

=

d

i

s

t

a

l

s

m

a

l

i

n

t

e

r

s

t

i

n

e

/

p

r

o

x

i

m

a

l

c

o

l

o

n

/

d

i

s

t

a

l

c

o

l

o

n

/

r

e

c

t

u

m

I

d

i

o

p

a

t

h

i

c

i

n

f

l

a

m

a

t

o

r

y

b

o

w

e

l

d

i

s

e

a

s

e

-

e

n

d

o

s

c

o

p

i

c

f

e

a

t

u

r

e

2

:

<

u

n

s

p

e

c

i

f

i

e

d

>

=

s

t

r

i

c

t

u

r

e

s

-

e

n

d

o

s

c

o

p

i

c

3

:

<

u

n

s

p

e

c

i

f

i

e

d

>

=

c

o

b

l

e

s

t

o

n

i

n

g

C

r

o

h

n

s

d

i

s

e

a

s

e

-

e

n

d

o

s

c

o

p

i

c

f

e

a

t

u

r

e

2

:

<

u

n

s

p

e

c

i

f

i

e

d

>

=

p

s

e

u

d

o

p

o

l

y

p

s

U

l

c

e

r

a

t

i

v

e

c

o

l

i

t

i

s

Transformation of Health Care Through AI Revolutions

50 of 129

Medical Imaging

Change Analysis – Lung Cancer
NIST/QIBA Activities
Iterative Reconstruction
Interpreting Wireless Capsule Endoscopy Images
From Images to Diagnosis through Ontologies
Image Quality for Healthcare Applications
Computational Metrology for Biomedical Imaging
Performance of Scalable Systems

50

Transformation of Health Care Through AI Revolutions

51 of 129

Confidence in CS Metrology & Big Data Measurements

51

Biological Metrology

Computational Science

-Experimental design

-Specimen preparation

-Visual inspection

-Definition of biologically meaningful objects & features

-Image correction

-Stitching

-Image visualization

-Segmentation & tracking & evaluation

-Feature extraction & visualization

-Data-driven hypotheses

-Manual labeling of samples

-Interpretation of measurements

-Decisions

-Comparisons & classification & cross-validation

-Confidence in models due to data variations and computational parameters

Courtesy: Peter Bajcsy and Mary Brady

Transformation of Health Care Through AI Revolutions

52 of 129

Clinical Environment: Cell Therapy for Age–Related Macular Degeneration (AMD) �[Stem Cell Engineering of Retinal Pigment Epithelium]

52

20/20 Vision

AMD blurred vision

6-9 months

Scientific insights: Quality assurance criteria for cell implants

11M in US

Courtesy: Peter Bajcsy

Global cost is $343 billion

Transformation of Health Care Through AI Revolutions

53 of 129

53

Scale

Nano to centimeter physical scale
TB- to PB-sized digital datasets

Complexity

Many instruments
Sample variety
Many models

Speed

Validate models
Verify computation
Explore and discover

Overarching Problem: Need Measurements from Images to Gain Scientific Insights

Transformation of Health Care Through AI Revolutions

54 of 129

IT/AI for Measurement Science

54

MEASUREMENT

MEASUREMENT TOOLS

One Large Field of View

Corrections

Stitching

Segmentation

Feature Extraction

AUTO-ANALYSES

Tracking

Re-Projection

Prediction Modeling

NETWORK

Image Pyramid

RAW DATA

NETWORK

BROWSER

Segmentation

Feature Extraction

Tracking

Sampling

Prediction Modeling

RAPID ANALYSES

Pan/Zoom
Image Features

Annotations
Predictions

https://isg.nist.gov/deepzoomweb/software/wipp

Transformation of Health Care Through AI Revolutions

55 of 129

Generative Adversarial Networks (GAN)

GANs are unsupervised learning models able to generate detailed realistic images.
They can be considered as a two-player game between a generator, which learns how to generate samples resembling real data, and a discriminator, which learns how to discriminate between real and generated data.

Step 1: Train the discriminator
Step 2: Train the generator via chained models

55

Real

Fake

Generator

Discriminator

Random noise

Fake or Real

Transformation of Health Care Through AI Revolutions

56 of 129

GAN Example Images

Examples of real/true and generated fluorescent and bright-field images of retinal pigment epithelial (RPE) cells

56

Transformation of Health Care Through AI Revolutions

57 of 129

Realistic Generation of Training Samples for Image Segmentation Using U-NET + GAN

All acquired images are passed through the GAN architecture so that the discriminator learns an abstract representation of the data without any supervision.
The discriminator weights are then transferred to the encoder part of U-Net segmentation model.
A small part of the images is manually segmented and used to train the U-Net.
During U-Net CNN model training, only the decoder weights are optimized.

57

Transformation of Health Care Through AI Revolutions

58 of 129

GENERATING RULES FROM NNS

58

Courtesy: Peter Bajcsy and Sarala Padi

Transformation of Health Care Through AI Revolutions

59 of 129

Accurate and Interpretable Classification of Microspectroscopy Pixels Using Artificial Neural Networks

59

Results

Context

Raman microspectroscopy and infrared (IR) absorption microspectroscopy are stain-free imaging techniques able to capture spectral information detailing the biochemical composition of cells and tissues.

Approach

Problem

The majority of chemical identification is usually performed visually by experts, capable of differentiating unique patterns of multiple peaks within the vibrational spectrum. There is a lack of precise mathematical formulations for the majority of existing expert labeling rules. Accurate automated spectral labeling together with discovery and material exploration are important aspects in the context of microspectroscopy.

Petre Manescu¹, Young Jong Lee², Charles Camp², Marcus Cicerone², Mary C. Brady¹, Peter Bajcsy¹

¹Software and Systems Division, Information Technology Laboratory National Institute of Standards and Technology Gaithersburg, MD 20899

²Biosystems and Biomaterials Division, Material Measurement Laboratory National Institute of Standards and Technology, Gaithersburg, MD 20899

Manescu et al. Accurate and Interpretable Classification of Microspectroscopy Pixels Using Artificial Neural Networks in Medical Image Analysis, (2017), Volume 37, pp 37-45

A new method to derive mathematical if-then decision rules for pixel classification from Artificial Neural Networks (ANN).

Transformation of Health Care Through AI Revolutions

60 of 129

60

Transformation of Health Care Through AI Revolutions

61 of 129

61

Transformation of Health Care Through AI Revolutions

62 of 129

62

Transformation of Health Care Through AI Revolutions

63 of 129

EVALUATING VARIOUS SCHEMES FOR NNS

63

Courtesy: Peter Bajcsy and Sarala Padi

Transformation of Health Care Through AI Revolutions

64 of 129

Modeling Approaches: Overview

64

Transformation of Health Care Through AI Revolutions

65 of 129

Quantitative Comparison of Modeling Approaches

65

Transformation of Health Care Through AI Revolutions

66 of 129

Qualitative Comparison of Modeling Approaches

66

Transformation of Health Care Through AI Revolutions

67 of 129

67

Transformation of Health Care Through AI Revolutions

68 of 129

Outline

AI: A Brief History
NIST and AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

68

Transformation of Health Care Through AI Revolutions

69 of 129

NNs Can be Fooled

69

Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Computer Vision and Pattern Recognition (CVPR ’15), IEEE, 2015.

Transformation of Health Care Through AI Revolutions

70 of 129

NNs Can be Fooled

70

Nguyen A, Yosinski J, Clune J. Deep Neural Networks are Easily Fooled: High Confidence Predictions for Unrecognizable Images. In Computer Vision and Pattern Recognition (CVPR ’15), IEEE, 2015.

Transformation of Health Care Through AI Revolutions

71 of 129

Natural Trojans

Background: An adversarial example is an input which causes an incorrect prediction.

Difference between a reliable adversarial example and a trojan is intention.

An adversarial example is a model training artifact which can be repurposed as a trojan.

https://openai.com/blog/adversarial-example-research/

Transformation of Health Care Through AI Revolutions

72 of 129

72

https://www.youtube.com/watch?v=8NNDCuSgls0

https://www.youtube.com/watch?v=-JEq8rMzkBs

Transformation of Health Care Through AI Revolutions

73 of 129

OKNs & NNs

Open Knowledge Networks and Category Theory
Terminology generation for OKNs
Predictive Coding

73

Transformation of Health Care Through AI Revolutions

74 of 129

Generic Information Modeling Requirements

Model construction
Representation across scales
Broad accommodation for multiple formalisms
Separation of domain-specific concerns
Integration and aggregation across models
Model evolution
Flexibility and modularity
Scalability

74

Transformation of Health Care Through AI Revolutions

75 of 129

Open Knowledge Networks (OKNs)

“The vision of OKN is to create an open knowledge graph of all known entities and their relationships, ranging from the macro (e.g., have there been unusual clusters of earthquakes in the US in the past six months?) to the micro (e.g., what is the best combination of chemotherapeutic drugs for a 56 y/o female with stage 3 brain cancer.) OKN is meant to be an inclusive, open, community activity resulting in a knowledge infrastructure that could facilitate and empower a host of applications and open new research avenues including how to create trustworthy knowledge networks/graphs” https://www.nitrd.gov/nitrdgroups/index.php?title=Open_Knowledge_Network.

75

See also: https://ontologforum.org/index.php/OntologySummit2020

Transformation of Health Care Through AI Revolutions

76 of 129

76

Ontology Spectrum

weak semantics

strong semantics

Is Disjoint Subclass of with transitivity property

Modal Logic

Logical Theory

Thesaurus

Has Narrower Meaning Than

Taxonomy

Is Sub-Classification of

Conceptual Model

Is Subclass of

DB Schemas, XML Schema

UML

First Order Logic

XML

ER

Relational Model, Extended ER

Description Logic

DAML+OIL, OWL

RDF/S

XTM

Syntactic Interoperability

Structural Interoperability

Semantic Interoperability

From less to more expressive

Courtesy: Leo Obrst, MITRE

Transformation of Health Care Through AI Revolutions

The Ontology Spectrum depicts a range of semantic models. What is normally known as an ontology can range from the simple notion of a the terminological model can range from the simple notion of a Taxonomy (terms[1] or concepts[2] with minimal hierarchic or parent/child structure), to a Thesaurus (terms, synonyms, broader than/narrower than term taxonomies, association relation), to a Conceptual Model (concepts structured in a subclass hierarchy, generalized relations, properties, attributes, instances), to a Logical Theory (elements of a Conceptual Model focusing however on real world semantics and extended with axioms and rules, also represented in a logical KR language enabling machine semantic interpretation). A Conceptual Model can be considered a weak ontology; a Logical Theory can be considered a strong ontology. The Ontology Spectrum therefore displays the range of models in terms of expressivity or richness of the semantics that the model can represent , from “weak” or less expressive semantics at the lower left (value set, for example), to “strong” or more expressive semantics at the upper right. The blue lines, labeled by syntactic interoperability, structural interoperability, and semantic interoperability, indicate roughly the expressiveness of the model required to respectively address those levels of interoperability. XML is sufficient for syntactic interoperability, XML Schema enables structural interoperability, but a minimum of RDF is necessary for semantic interoperability.

�[1] Terms (terminology): Natural language words or phrases that act as indices to the underlying meaning, i.e., the concept (or composition of concepts) The syntax (e.g., string) that stands in for or is used to indicate the semantics (meaning).

[2] Concept: A unit of semantics (meaning), the node (entity) or link (relation) in the mental or knowledge representation model. In an ontology, a concept is the primary knowledge construct, typically a class, relation, property, or attribute, generally associated with or characterized by logical rules. In an ontology, these classes, relations, properties are called concepts because it is intended that they correspond to the mental concepts that human beings have when they understand a particular body of knowledge (subject matter area or domain). In general, a concept can be considered a placeholder for a real world referent, and thus ontology as an engineering prduct is about representing the semantics of the real world in a model that is usable and interpretable by machine.

77 of 129

Toward A Calculus of Information

In mechanical engineering, Newton’s calculus provides:

A language for representing mechanical states, processes and behavior.
A collection of standard techniques and algorithms for analyzing mechanical systems expressed in this language.

Predicate calculus = language & techniques for logic.

λ-calculus = language & techniques for computation.

Category Theory = language & techniques for composition.

- Generalizes all three calculi above

Formal representation in CT provides “hygiene”, helping to guide our thinking and to avoid errors and misrepresentation.

77

Transformation of Health Care Through AI Revolutions

78 of 129

Virtues of CT Modeling

78

Modeling Requirements	Associated CT Construction
Model construction	Free completion
Multiple formalisms	Semantic categories
Domain-specificity	Forgetful functors
Integration	Compositionality
Evolution	Kan extension
Modularity	Colimits
Scalability	Functional programming
Representation across scales	Colimits + Functors

Transformation of Health Care Through AI Revolutions

79 of 129

79

Ontology Spectrum

weak semantics

strong semantics

Is Disjoint Subclass of with transitivity property

Modal Logic

Logical Theory

Thesaurus

Has Narrower Meaning Than

Taxonomy

Is Sub-Classification of

Conceptual Model

Is Subclass of

DB Schemas, XML Schema

UML

First Order Logic

XML

ER

Relational Model, Extended ER

Description Logic

DAML+OIL, OWL

RDF/S

XTM

Syntactic Interoperability

Structural Interoperability

Semantic Interoperability

From less to more expressive

Courtesy: Leo Obrst, MITRE

CATEGORY THEORY

Transformation of Health Care Through AI Revolutions

The Ontology Spectrum depicts a range of semantic models. What is normally known as an ontology can range from the simple notion of a the terminological model can range from the simple notion of a Taxonomy (terms[1] or concepts[2] with minimal hierarchic or parent/child structure), to a Thesaurus (terms, synonyms, broader than/narrower than term taxonomies, association relation), to a Conceptual Model (concepts structured in a subclass hierarchy, generalized relations, properties, attributes, instances), to a Logical Theory (elements of a Conceptual Model focusing however on real world semantics and extended with axioms and rules, also represented in a logical KR language enabling machine semantic interpretation). A Conceptual Model can be considered a weak ontology; a Logical Theory can be considered a strong ontology. The Ontology Spectrum therefore displays the range of models in terms of expressivity or richness of the semantics that the model can represent , from “weak” or less expressive semantics at the lower left (value set, for example), to “strong” or more expressive semantics at the upper right. The blue lines, labeled by syntactic interoperability, structural interoperability, and semantic interoperability, indicate roughly the expressiveness of the model required to respectively address those levels of interoperability. XML is sufficient for syntactic interoperability, XML Schema enables structural interoperability, but a minimum of RDF is necessary for semantic interoperability.

�[1] Terms (terminology): Natural language words or phrases that act as indices to the underlying meaning, i.e., the concept (or composition of concepts) The syntax (e.g., string) that stands in for or is used to indicate the semantics (meaning).

[2] Concept: A unit of semantics (meaning), the node (entity) or link (relation) in the mental or knowledge representation model. In an ontology, a concept is the primary knowledge construct, typically a class, relation, property, or attribute, generally associated with or characterized by logical rules. In an ontology, these classes, relations, properties are called concepts because it is intended that they correspond to the mental concepts that human beings have when they understand a particular body of knowledge (subject matter area or domain). In general, a concept can be considered a placeholder for a real world referent, and thus ontology as an engineering prduct is about representing the semantics of the real world in a model that is usable and interpretable by machine.

80 of 129

OKNs & NNs

Open Knowledge Networks and Category Theory
Terminology generation for OKNs
Predictive Coding

80

Transformation of Health Care Through AI Revolutions

81 of 129

A System for Automatically and Adaptably Building Indexes and Terms from Very Large Technical Document Collections�Using Linguistic Structure and Deep Learning�

Talapady N. Bhat, Jacob N. Collard, Ira A. Monarch,

Sarala Padi, Ananya Srinivasan, Ram D. Sriram, Eswaran Subrahmanian

81

Transformation of Health Care Through AI Revolutions

82 of 129

The Approach

Bottom up natural language analysis based on usage
Insights from glutinous languages (such a Sanskrit and other Indo-European languages)
English is a limited non glutinous language: ex. Policeman but not Police Dog.
Use conjugation of nouns into compound noun structure using phrases and specific rules
Semi-automated approach to Taxonomy of Terms

82

Transformation of Health Care Through AI Revolutions

83 of 129

Insight from Sanskrit as an Exemplar for Our Root and Rule method

Compounding nouns originally used for epiphonic purposes
Economy of expression and memory
First encoded rules of compounding of terms in Sanskrit - Panini (see https://web.stanford.edu/class/linguist289/encyclopaedia001.pdf)
The question is can we use this insight to create compounded terms in English through

Natural language extraction of non phrases
Design of Rules for conjugation

Benefit would be encode root term and its qualifiers to preserve context of use.

83

Transformation of Health Care Through AI Revolutions

84 of 129

Normalizing syntax

84

B

This process creates the term: “Inertial reference frame”

“Frame of inertial reference” will also get converted to same Term

Transformation of Health Care Through AI Revolutions

85 of 129

R&R in Use

Taxonomy Creation
Search
Document Clustering

85

Transformation of Health Care Through AI Revolutions

86 of 129

Taxonomy Fragment Created by R&R

86

Diabetes-patient

Elderly-diabetes-patient

Cerebral-ischemia-:suffer:-elderly-diabetes-patient

Mellitus-diabetes-patient

Hypertension-mellitus-diabetes-patient

cataract:-mellitus:-diabetes-patient

Malnutrition-mellitus-diabetes-patient

ESRD-cause-mellitus-diabetes-patient

Transformation of Health Care Through AI Revolutions

87 of 129

R&R in Use

Taxonomy Creation
Search
Document Clustering

87

Transformation of Health Care Through AI Revolutions

88 of 129

NIST Covid Data repository

88

Transformation of Health Care Through AI Revolutions

89 of 129

R&R Embedded in Covid Data Registry

89

Transformation of Health Care Through AI Revolutions

90 of 129

Search Results

90

Transformation of Health Care Through AI Revolutions

91 of 129

OKNs & NNs

Open Knowledge Networks and Category Theory
Terminology generation for OKNs
Predictive Coding

91

Transformation of Health Care Through AI Revolutions

92 of 129

Meeting in the Middle

OKNs’ strengths balance NNs’ weaknesses, and vice versa:

92

OKN	NN
Unstructured data	Identifying patterns
Background knowledge	Training Data
Complex Logic	Explanation
Elicitation	Online learning

Courtesy: Spencer Breiner, Eswaran Subrahmanian

Transformation of Health Care Through AI Revolutions

93 of 129

Predictive Coding (PC)

93

Courtesy: Spencer Breiner, Eswaran Subrahmanian

Transformation of Health Care Through AI Revolutions

94 of 129

OKNs and NN meet Through PC

Ontologies generate probabilistic models in (at least) two ways:

Elicited and encoded directly (e.g., medical ontologies).
Calculated from structured data (e.g., manufacturing ontology & database).

PC learning algorithm updates both models together, linking black-box NN parameters to explainable OKN values.

94

NN

Data

Courtesy: Spencer Breiner, Eswaran Subrahmanian

OKN

Transformation of Health Care Through AI Revolutions

95 of 129

95

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3336915/

Transformation of Health Care Through AI Revolutions

96 of 129

Outline

AI: A Brief History
NIST and AI
Health Care and AI
Rise of KBS and First Wave: Mycin/Emycin & Explanations
Second Wave: CNNs & Explanations
Third Wave: OKNs & NNs
Future & Summary

96

Transformation of Health Care Through AI Revolutions

97 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Emotion Recognition
Toward Smart Health Care
Summary

97

Transformation of Health Care Through AI Revolutions

98 of 129

98

Transformation of Health Care Through AI Revolutions

99 of 129

National AI R&D Strategic Plan (2019)

Strategy 1: Make long-term investments in AI research. Prioritize investments in the next generation of AI that will drive discovery and insight and enable the United States to remain a world leader in AI.
Strategy 2: Develop effective methods for human-AI collaboration. Increase understanding of how to create AI systems that effectively complement and augment human capabilities.
Strategy 3: Understand and address the ethical, legal, and societal implications of AI. Research AI systems that incorporate ethical, legal, and societal concerns through technical mechanisms.
Strategy 4: Ensure the safety and security of AI systems. Advance knowledge of how to design AI systems that are reliable, dependable, safe, and trustworthy.
Strategy 5: Develop shared public datasets and environments for AI training and testing. Develop and enable access to high-quality datasets and environments, as well as to testing and training resources.
Strategy 6: Measure and evaluate AI technologies through standards and benchmarks. Develop a broad spectrum of evaluative techniques for AI, including technical standards and benchmarks.
Strategy 7: Better understand the national AI R&D workforce needs. Improve opportunities for R&D workforce development to strategically foster an AI-ready workforce.
Strategy 8: Expand public-private partnerships to accelerate advances in AI. Promote opportunities for sustained investment in AI R&D and for transitioning advances into practical capabilities, in collaboration with academia, industry, international partners, and other non-Federal entities.

99

Transformation of Health Care Through AI Revolutions

100 of 129

100

Transformation of Health Care Through AI Revolutions

101 of 129

101

Transformation of Health Care Through AI Revolutions

102 of 129

102

Transformation of Health Care Through AI Revolutions

103 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Emotion Recognition
Toward Smart Health Care
Summary

103

Transformation of Health Care Through AI Revolutions

104 of 129

Performance vs. Explainability: DARPA XAI Program

104

Explainability

Learning Techniques (today)

Explainability (notional)

Neural Nets

Statistical Models

Ensemble Methods

Decision Trees

Deep Learning

SVMs

AOGs

Markov Models

MLNs

Bayesian Belief Nets

SRL

CRFs HBNs

Random Forests

Graphical Models

Learning Performance

Transformation of Health Care Through AI Revolutions

105 of 129

Performance vs. Explainability: DARPA XAI Program

105

Explainability

Learning Techniques (today)

Explainability (notional)

Neural Nets

Statistical Models

Ensemble Methods

Decision Trees

Deep Learning

SVMs

AOGs

Markov Models

MLNs

Model Induction

Techniques to infer an explainable model from any model as a black box

Deep Explanation

Modified deep learning techniques to learn explainable features

XAI Approach

Create a suite of machine learning techniques that produce more explainable models, while maintaining a high level of learning performance

Interpretable Models

Techniques to learn more structured, interpretable, causal models

Bayesian Belief Nets

SRL

CRFs HBNs

Random Forests

Graphical Models

Learning Performance

Transformation of Health Care Through AI Revolutions

106 of 129

What XAI is Trying To Do?

106

Learning Process

Training Data

Learned Function

Output

Today

This is a cat

(p = .93)

Why did you do that?
Why not something else?
When do you succeed?
When do you fail?
When can I trust you?
How do I correct an error?

User with a Task

Training Data

New Learning Process

Explainable Model

Explanation Interface

Tomorrow

I understand why
I understand why not
I know when you’ll succeed
I know when you’ll fail
I know when to trust you
I know why you erred

This is a cat:

It has fur, whiskers, and claws.
It has this feature:

User with a Task

http://explainthatstuff.com

Transformation of Health Care Through AI Revolutions

107 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Emotion Recognition
Toward Smart Health Care
Summary

107

Transformation of Health Care Through AI Revolutions

108 of 129

Can you Trust Dr. AI?

108

https://www.nasw.org/article/doctors-and-engineers-are-asking-can-we-trust-dr-ai

https://drive.google.com/file/d/1tX2SjjGQQNp3uJ5kpwNJfGPW-b40jE7H/view

AAAS Symposium 2021, February

Transformation of Health Care Through AI Revolutions

109 of 129

TrojAI

Problem:

Adversarial AI attacks insert trojan triggers that alter the behavior of the system.
Modern AI advances are characterized by vast, crowdsourced datasets that are impractical to clean or monitor.
Developers may ingest trojan triggers through the process of transfer learning.

Goal: Identify trojan triggers prior to widespread deployment

Transformation of Health Care Through AI Revolutions

AI is no longer a science fiction dream, it is now just another tool in the software toolbox.

The problem is, in becoming commercial off the shelf products, the black box nature of AI can be leveraged to hide malicious behavior.

There are two broad attack surfaces for inserting trojans into AI models.

First you can attack the dataset. AI is powered by vast datasets which are impractical to fully clean and vet. So one can inject malicious samples into the dataset upstream of the training process.

Second, you can directly attack the training process itself. Model zoos allow individuals to publish pre-trained AIs with little to no provenance information.

This allows anyone to upload an AI with potential backdoors trained into it.

Additionally, trojans have been shown to survive transfer learning, so refining the model on your data will not guarantee it is clean.

The goal of the TrojAI program is to foster research & development into methods for detecting poisoned AI models before deployment.

With the emergence of AI models for end to end data processing, the old problem of computer viruses is coming back.

However, unlike traditional software, you cannot perform a code review on software 2.0 where the core logic is within an AI model’s weights.

Therefore, you need new methods for determining whether the code/models you are using have not been compromised.

The canonical example of a trojan is a self driving car mis-classifying a stop sign as a speed limit sign due to a post-it-note on the sign.

110 of 129

Motivation

Cultivate trust in�

Trojan detectors

Define/test metrics using data available from TrojAI program �

Trojan detector deployments

Evaluate/analyze detector performance for different deployment strategies�

AI design

Investigate correlation between trojan detection and model parameters

Transformation of Health Care Through AI Revolutions

111 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Emotion Recognition
Toward Smart Health Care
Summary

111

Transformation of Health Care Through AI Revolutions

112 of 129

Emotion recognition (ER)

It plays a key role in human-computer interaction
It can enrich the next-generation AI with emotional intelligence

Applications:

Customer/representative behavior analysis
Call center services
Gaming
Personal assistants
Health care
Manufacturing: Alerting the driver
…

Modalities in which humans' express emotions

Speech, Text, Video, Facial expressions, Image, Hand gestures, etc.

Emotion Recognition

Courtesy: Sarala Padi

Transformation of Health Care Through AI Revolutions

113 of 129

Health care providers use AI to

Prioritize patients by analyzing their facial expressions

Patients who are in most discomfort could receive highest priority

Telemedicine

To understand patients feelings without their physical presence

Modalities:

Electroencephalography (EEG)
Electrocardiography (ECG)
Respiration
Gesture

Challenges:

Emotion recognition accuracy of AI models is a big concern
Training AI models require large amount of data
Facial expressions are culturally variable

Diverse training of AI system is hard

Emotion Recognition in Healthcare

Transformation of Health Care Through AI Revolutions

114 of 129

Traditional machine learning

Based on bag of engineered features (e.g., MFCC, ZCR, pitch, entropy)
Performance dependent on the type and diversity of features
Unclear which features correlate most with various emotions
Research is still in progress to explore additional features

Deep neural networks

End-to-end model and directly extracts the features from the spectrogram
Benefit from the multiple modalities by extracting complementary information

To address data scarcity

Multiwindow augmentation
Transfer learning
Spectrogram augmentation

Artificial Intelligence for Emotion Recognition

Transformation of Health Care Through AI Revolutions

115 of 129

Unimodal: Speech

Augmentation:

Multi-window data augmentation:

Windowing is applied at different scales to extract features to augment the AI model

Spectrogram Augmentation:

Time and frequency masking were applied to generate additional data samples to train AI model

Transfer learning:

Model built for speaker recognition task is adapted for emotion analysis

Multimodal: Speech and Text

Speech: Transfer learning to adapt speaker recognition models for ER analysis
Text: Transformer based embeddings for ER analysis
Fusing the scores from speech and text modalities to improve the ER accuracy

AI for Emotion Recognition Task

Future work:

Building the multimodal ER models by including videos and facial landmarks
Evaluating the Multimodal AI systems across datasets

Transformation of Health Care Through AI Revolutions

116 of 129

Multiwindow

Transfer learning

Multimodal Emotion Recognition

With Sarala Padi, Dinesh Manocha & Omid Sadjadi

Transformation of Health Care Through AI Revolutions

117 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Emotion Recognition
Toward Smart Health Care
Summary

117

Transformation of Health Care Through AI Revolutions

118 of 129

118

Environment

Physical World

Physical Network

Things

Society

Formal Social Organizations

Informal Social Organizations

Interconnected Systems & Control

Sensing and Acting

Internet of Things

Person

Cyber Physical Systems

Cyber Physical Human Systems

Cyber Organizational Networks

Social Sensing and Acting

Internet of People

Cyber Social Networks

Interconnected Social Networks

Smart Networked Systems and Societies

Interconnected Cyber Physical and Social Networks

Interconnected Formal Social Organizations

Internet of

Formal Social Organizations

Sensing and Acting

Transformation of Health Care Through AI Revolutions

119 of 129

Defining Health Persona

119

HEALTH PERSONA

Logical Sensor

Physiological Sensors

Fitness Tracking Sensors

Life Event

Kinetic Event

Physiological Event

Food Event

Personicle

Calendar

Activity-level

Heart rate

Home

Drive

Meeting

Work

Exercise

Walk

Courtesy: Ramesh Jain, UCI

Transformation of Health Care Through AI Revolutions

120 of 129

Personal Health Record

120

Electronic Health Record

Genomic

Analysis

HEALTH PERSONA

+

PHR

CYP2C19

Plavix?

Transformation of Health Care Through AI Revolutions

121 of 129

121

EventShop : Global Situation Detection

Predictive Situation Recognition

Evolving Global Situation

Predictive Personal Situation Recognition

Personal EventShop

Evolving Personal Situation

Need- Resource Matcher

Recommendation Engine

Persona

Database

Resources

Needs

Data Ingestion

Wearable

Sensors

Calendar

Location

….

Data Sources

Data Ingestion and aggregation

Database Systems

Satellite

Environmental Sensor Devices

Social Network

Internet of Things

Actionable Information

Courtesy: Ramesh Jain, UCI

Example of Physical Data
AQI: 250 (Air Quality Index 0<AQI<500) at Location DC
GPS: 38, 53, 77, 02, 12:00
Temperature: 60 F

- Location identification DC / Map Visualization at time 2:30 pm

Example of Rules

R1: If (301<AQI<500 at Location DC ) Then (“Health alert”: everyone may experience more serious health effects) http://www.airnow.gov/
R2: If (Disease = “Asthma” and AQI>200) Then (message…)
…

- Event: Breathing/cough symptom

- Posted by: a patient

- Where: DC

- When: 2/21/13 ç2:30 am

Prone to Asthmatic reactions

Is in outside location DC at time 2:30 pm

High Probability of attack

Air quality not suitable for your Asthma – Move-Indoors

Transformation of Health Care Through AI Revolutions

122 of 129

Courtesy: Fred Hosea, Ph.D.�Program Director�Clinical Technology

fred.w.hosea@kp.org

Smart Healthcare

Smart Devices
Smart Networks
Smart Processes
Smart EMRs
Smart Medicine
Smart Organizations
Smart Collaborations
Smart Society
Smart Planet

Transformation of Health Care Through AI Revolutions

123 of 129

The 21^st Century Doc

123

Courtesy: Rod Grupen & Tihomir Latinovic

Transformation of Health Care Through AI Revolutions

124 of 129

124

Zetta Science

The

Fifth Paradigm

Web & Distributed Innovation

Singularity Press

The

Sixth Paradigm

Knowledge & Visualization

The World Press

The

Seventh Paradigm

Smart Networked Systems & Societies

Transformation of Health Care Through AI Revolutions

125 of 129

125

Transformation of Health Care Through AI Revolutions

126 of 129

Future & Summary

Strategic plans
XAI: Explanation-based Learning
Can you trust Dr. AI?
Toward Smart Health Care
Summary

126

Transformation of Health Care Through AI Revolutions

127 of 129

Summary

Metrology for AI and AI for Metrology
Standards introduced at the right time will lead to innovation
Testing algorithms will increase trust in AI
Learning programs need to explain reasoning
Public-Private partnerships may help accelerate progress
Challenge problems will improve performance

127

Transformation of Health Care Through AI Revolutions

128 of 129

Acknowledgments and Disclaimer

This talk is based on the work by several people at NIST. Credit is provided in appropriate slides. My thanks to all those who provided the slides.
Commercial equipment and software, many of which are either registered or trademarked, are identified in order to adequately specify certain procedures. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose.

128

Transformation of Health Care Through AI Revolutions

129 of 129

In Remembrance: Sargur Srihari �(1949-05-07 to 2022-03-08)

129

Transformation of Health Care Through AI Revolutions