1 of 16

Securing Embold Health’s�AI-powered product

2 of 16

Embold’s mission

Helping people find the right doctor. The first time. ��We drive better health outcomes and cost savings using our industry leading physician-level quality analytics and interactive member tools to connect people to top-performing physicians in their local communities.

2

3 of 16

Low-quality care hurts

3

Across 30,000 members,

13,500

or 45%

will look for a new doctor each year.

Of those members,

4,860

or 16%

will unknowingly choose low-performing doctors.

Avoidable hospitalization

Unnecessary surgery

Higher complication rates

Unplanned time off work

Results of Low-Quality Care:

4 of 16

Data and access

4

5 of 16

Business problems and solutions

How do we assist end-users if they do not know what kind of provider they may need?��Can we utilize generative AI to assist end-users in finding a provider in a secure and responsible manner?��Solution: Azure OpenAI?

5

Can we rate providers on the appropriateness, effectiveness and the cost of care?��If so, how do we deliver provider focused quality metrics to end-users?��Solution: Integrations, data feeds and Provider Guide

Original

New

6 of 16

Artificial intelligence risks

6

Sensitive data disclosure

Unauthorized access

Intellectual property leaks

Compliance violations

Inaccurate AI outputs

Biased decision-making

Ethical concerns

System integration issues

Misinterpretation of outputs

Misuse of AI tools

Lack of transparency

Legal liabilities

Data poisoning attacks

Deepfake creation risks

Disinformation spread

Dependency on AI

Disallowed content generation

Defamation

Denial of accountability

Delay in responses

Data integrity loss

Development delays

Distortion of information

Damage to reputation

Divulgence of secrets

Disruption of workflows

Disconnect with users

Diminished user trust

Decreased control

Disruption of services

Accountability gaps

Uncertainty begins here.

7 of 16

Artificial intelligence controls

7

Policy and governing documentation created. Pulls from HITRUST, IAPP, NIST and ISO.

Policy
Procedure
Product Specific Documentation
Existing policy and procedure alignment
Quality Management

Appropriateness (Emergency vs. Non-emergency)

Effectiveness

Repeatability

Uncertainty continues here.

8 of 16

Artificial intelligence governance

8

Responsible AI

Ethical Use
Transparency
Accountability

Human Oversight

Supervision
Intervention
Alignment

Quality Measures

Appropriateness
Effectiveness
Repeatability

Uncertainty continues here.

Responsible AI

Ethical Use - The commitment to using AI in a manner that is ethical, non-discriminatory, safe and beneficial to society.
Transparency - Fully transparent for all documentation, sign-off, risks, etc. Communicate deficiencies, deviations in quality, and human oversight.
Accountability - Clearly define accountability for AI outcomes, ensuring responsible use and management.

Human Oversight

Supervision - Implement regular human supervision to verify AI outputs for accuracy and ethical compliance. Quarterly reviews by clinical review board, quality team, product owner coupled with elements of quality management.
Intervention - Establish clear protocols for human intervention in cases where AI behavior deviates from expected norms. Figure out baseline acceptable % and come up with plan for remediation when deviations happen to bring levels back to acceptable. This will happen during the SDLC of pre-production testing and acceptance.
Alignment - Alignment of the AI system project-level processes with the organization’s objectives

Quality Standards

Appropriateness - Regularly assess AI outputs for alignment with intended objectives and company standards.
Effectiveness - Monitor the efficiency of AI in achieving objectives within reasonable timeframes or interactions.
Repeatability - Ensure consistent, reliable results from AI under similar conditions. Automated and manual testing of specific scenarios are completed quarterly

9 of 16

Artificial intelligence governance

9

Security

Protection
Consequence �Management
Advisement

Privacy

Notice
Consent
Participation

Legal

Contractual
Due Diligence
Federal and state law

Uncertainty continues here.

10 of 16

We interrupt this story to give a shoutout

10

HITRUST Cybersecurity Certification for Deployed AI Systems

AI security requirements included

AI security threat management
AI security governance and oversight
Development of AI software
AI legal and compliance
AI supply chain
Model robustness
Access to the AI system
Encryption of AI assets
AI system logging and monitoring

AI security threats considered

Availability attacks
Input-based attacks
Poisoning attacks
Supply chain attacks
Threats inherent to language models

11 of 16

Artificial intelligence security

11

Can I phone a friend?

Uncertainty ends here.

12 of 16

StackAware helps AI-powered companies manage their risk related to:

Cybersecurity
Compliance
Privacy

12

13 of 16

Actions taken

AI risk assessment

Review of:

Vendors
Terms of use
Privacy notice
Data classification
Existing policies and procedures

Deliverables

Executive-level presentation
Detailed risk register

AI penetration test

Technical validation of application security.

Leveraged StackAware AI penetration test procedure to review for both generative AI and underlying infrastructure vulnerabilities.

Report provided quantitative assessments about likelihood of exploitation (not “high” or “low”).

13

14 of 16

Lessons learned

AI governance starts with asset inventory and data classification.

Validation of inputs - and guardrails to handle unexpected ones - are key when designing generative AI applications

Non-technical risks can be just as - if not more - important than technical ones. And harder to identify.

14

15 of 16

Initial Testing Results

15

	Precision	Recall	F1 �Weighted	Observations
Emergency	99.60%	99.43%	99.55%	3,884
Specialty	95.53%	98.66%	97.07%	5,841
Global �Summary	97.19%	98.98%	98.07%	9,723

16 of 16

Questions?

Walter Haydock, CEO

Steve Dufour, CSO/CPO