Scale Up Interoperability and Assurance with InCommon Federation Expectations

December 9, 2025

- Community Trust and Assurance Board (CTAB)�- Technical Advisory Committee

• 2025 Progress Report from CTAB and TAC
• SIRTFI Exercise 2025
• Identity Assurance Deployment Guidance
• Subject Identifiers Deployment Guidance
• REFEDS Access Entity Categories
• REFEDS MFA Profile 2.0
• Federation Expectation Program
• Community needs and benefits
• Principles
• Program details
• Projected timeline and cadence

Agenda

| 2

InCommon Technical Advisory Committee�

Community Trust and Assurance Board�

Keith Wessel, University of Illinois Urbana-Champaign

Joanne Boomer, University of Missouri

Jeffrey Crawford, University of California, San Francisco

Matthew Economou, Independent

Derek Eiler, University of Nevada System

Björn Mattsson, Sunet

Andrew Morgan, Oregon State University

Steven Premeau, Independent

Mark Rank, Cirrus Identity

Jim VanLandeghem, Moran Technology

Marina Krenz, REN-ISAC

David Walker, Independent

Eric Goodman, Independent

Grady Bailey, Internet2

David Bantz, University of Alaska

Jon Miner, University of Wisconsin-Madison

Warren Anderson, LIGO

Pål Axelsson, SUNET

Matthew Eisenberg, National Institutes of Health

Richard Frovarp, North Dakota State University

Michael Grady, Unicon

Scott Green, Eastern Washington University

Christopher Keith, Brown University

Kyle Lewis, Research Data and Communications Technologies

Ryan McDaniel, University of Alaska Anchorage

Rick Wagner, Argonne National Laboratory

Gabor Eszes, University of Virginia

Tom Barton, Internet2

| 3

SIRTFI Exercise 2025

| 4

2025: InCommon’s fourth annual�Cybersecurity Cooperation Exercise

Sirtfi is part of InCommon Baseline Expectations, but…

​

• How many security teams have email published to metadata that are unaware of InCommon?
• How many security team personnel on a given day, over time, know about Sirtfi?
• How many institutional processes have been updated to identify multi-participant events and know how to use Sirtfi to bring response teams together?

​

This annual event helps build practical awareness among the ”teams on the ground”.

Cybersecurity Cooperation Exercise�What is it?

• Week-long event -- Distributed Narrative Tabletop
• Learning Objectives: Participants:
• Respond from other team’s inputs into their real-world published security contacts
• Using entity-IDs from login events, find real-world security contact of next playing team
• Member Focused (not federation operator focused)
• Minimum Requirements to participate: SP or IdP in Sirtfi framework
• International eduGain IdPs and SPs welcome
• Custom story-based scenario tailored to participant organizations each year.

​

​

​

SkaiNet

(Cirrus Identity)

Clairity

(NIAID)

OpenSkai

(Rice U)

OpenSkai LLM Dev Team

SkaiNet Investigator Team (AI Research)

SkaiNet Research

Data Manager

Clairity LLM Dev Team

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

UNC – Chapel Hill

Caroline Sweet

Real Diamond

Role-based

Access Model

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

API + Chat Page access

App to App API access

App Dev Access

LLM Training and Dev Access

Programmatic Mgmt

Access (Administrative)

U of Alaska

Salmon Floyd

EXERCISE EXERCISE EXERCISE

Source IPs:

X.X.X.X (Cambodia/Burma)

Y.Y.Y.Y (Indonesia)

2025’s Scenario

What Would Villains Do?

SAR’s Goals and Objectives

Goal: Discredit AI use by creating catastrophic outcomes that forces society to turn away from AI

​

Strategy: Compromise SkaiNet Investigators and LLM Developers for a Two-Pronged attack

​

• Compromised Investigator accounts to alter research observations: reduce observed hallucinations making models look more reliable
• Compromise LLM Dev accounts to increase LLM hallucination and error rates

​

SAR doesn’t care which model is chosen. They care that the Govt believes either model is safe to use and adopts it for cases with real threat to human life (medical use), while simultaneously making each model worse in outcomes. This would lead to loss of health or even loss of life in patient outcomes. Eventually, the AI would be blamed.

​

SAR will sacrifice visibility of PI account compromise and fake account created believing that investigation will stop after those are secured… they hope to continue to enjoy the other investigator and dev access at other universities.

​

They weren’t counting on Sirtfi…

​

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

Clairity LLM Dev Team

SAR recruits Caroline from IdP2 (grad student insider threat).

​

Caroline submits request form to PI at IdP1 with pdf trojan.

Compromises Dr Grant’s Account

Timeline: 28 days (20-24 Oct)

OpenSkai LLM Dev Team

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

UNC – Chapel Hill

Caroline Sweet

Real Diamond

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

SkaiNet Investigator/AppDev Team

Research

Data Manager

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

Upon reviewing request form, Dr. Grant initiates an account request with IdP2 and forwards the form to his Data Manager at IdP4.

Timeline: 28 days (20-24 Oct)

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

UNC – Chapel Hill

Caroline Sweet

Real Diamond

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

IdP2 creates Real Diamond’s account and provides credentials to Caroline the grad student.

​

Salmon Floyd receives the request and logs into SP1 to initiate the user profile and access permissions workflow for Real Diamond.

​

PDF trojan harvests Salmon Floyd’s password, but SAR can’t log into SP1 due to MFA.

Timeline: X-21 days (21-31 Oct)

Logs will show routine logins from known IPs even for compromised accounts at this point.

​

X.X.X.X login attempt @SP1, but due to no second factor, SAR can’t access SP1�2025-10-25T01:30

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

UNC – Chapel Hill

Caroline Sweet

Real Diamond

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

Caroline Provides SAR with Real Diamond’s credentials, and SAR start accessing SP1 and SP2,

Interacting with SP2 through SP1 prototype apps, and directly via SP2’s web portal to their chatbot.

​

SAR only has read access at this point; no write access to the model itself. Their aim is to poison the LLM and the research data, but they haven’t gotten far enough yet.

​

Right now SAR is just playing around with the app, API key, chat, and getting user lists to start building the social network profile for exploitation.

​

Timeline: X-21 days (27-31 Oct)

From X.X.X.X

SP1: Real.Diamond downloads user lists

Downloaded lists of users; 2025-10-28T14:30

​

SP2: Real Diamond: LLM chat usage and accesses app at admin level: downloads logs of user access: 2025-10-29T13:00

​

Also uses API key (which won’t show up in IdP2’s logs once established, but SP2 will see it, but this time from Y.Y.Y.Y 2025-10-30T02:00

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

UNC – Chapel Hill

Caroline Sweet

Real Diamond

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

SAR manages to get Salmon’s MFA credential through MFA fatigue, login, and disable rqmt for MFA. They now have Salmon’s access to change user permissions. They elevate Real Diamon’s access to SP1. Uses Real Diamond to download user registries from SP1 and SP2.

�SAR starts using second IP address range expecting first to be blocked in the aftermath.

​

Timeline: X-14 days (3-7 Nov)

Logs;

Salmon access SP2 from X.X.X.X, 2025-11-3T12:30

​

SP1 from Y.Y.Y.Y 2025-11-3T13:00

​

SP1 and SP2 access from X.X.X.X by Salmon Floyd to update Real Diamond’s access in SP1 and SP2

​

IdP2 Real Diamond Access at SP1 from X.X.X.X; app use, and download of full user team account registry and project plans (including data management and data storage plans) 2025-11-5T02:00

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

UNC – Chapel Hill

Caroline Sweet

Real Diamond

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

Dr. Grant goes on holiday. While out, SAR uses access to phish Users 3,5,6,7,8,9,10

​

These users go to a site that looks like a survey related to the study.

​

Timeline: X-14 days (3-7 Nov)

Logs;

None in the federation IAM stack.

​

Emails with links to convincing site

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

UNC – Chapel Hill

Caroline Sweet

Real Diamond

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

SAR now uses the research teams to increase hallucinations in the LLMs and hide evidence of hallucination in the observation data.

Timeline: X-7 days (10-14 Nov 2025)

SP Logs (IdP logs need to correspond);

SP1: Arty Fishal from X.X.X.X; alters research data; 2025-11-10T00:30

SP1: Kent Loggin from X.X.X.X downloads data 2025-11-9:T18:00

SP1: Anette Work from X.X.X.X changes her own research observations to reduce reports of hallucinations 2025-11-10:T06:00

SP1: Ellie Vate logs into SP1 from Y.Y.Y.Y, accesses apps 2025-11-11:T08:30

SP1: Lee King logs into SP1 from Y.Y.Y.Y, accesses apps 2025-11-11:T09:00

SP1: Sooper User logs into SP1 from Y.Y.Y.Y, accesses apps 2025-11-11:T10:00

SP1: Cody McCoderson accessing SP1 from Y.Y.Y.Y 2025-11-11:T10:30

​

SP2:

Arty Fishal poisoning LLM weights from X.X.X.X 2025-11-12T13:00 then Y.Y.Y.Y 2025-11-12T23:00

Cody McCoderson from X.X.X.X poisoning weights in SP2 2025-11-11T14:00

​

​

SP3: Kent Login from Y.Y.Y.Y chats with SP3 2025-11-10T16:00

SP3: Ellie Vate logs into SP3 from Y.Y.Y.Y, poisons weights 2025-11-12T13:00

SP3: Lee King logs into SP3 from Y.Y.Y.Y, poisons weights 2025-11-12T13:30

SP3: Sooper User logs into SP3 from Y.Y.Y.Y, poisons weights 2025-11-14T23:30

SP3: Anette Work logs into chat from Y.Y.Y.Y 2025-11-10:T05:00

​

​

​

​

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U of Missouri

Cody McCoderson

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

UNC – Chapel Hill

Caroline Sweet

Real Diamond

SP1

SkaiNet

SP2

Clairity

SP3

OpenSkai

OpenSkai �LLM Dev Team

Clairity LLM Dev Team

SAR now feels their position is solid and will continue.

​

To troll, they have Grant’s account deface the website, which is reported to SP1.

​

They expect Real Diamond will be discovered and want it to be a diversion to continue enjoying access to all the other users.

​

Little do they know of Sirtfi.

​

​

​

Timeline: X day: (17 Nov 2025)

SP2 logs Needsa Grant from X.X.X.X logging into chat LLM web app 2025-11-17T0530

​

​

SP1 logs Needsa Grant from X.X.X.X. changing web announcement

2025-11-17T06:00

​

​

​

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

Medical application modeling�comparing two LLMs

LLM Claiming to�have solved hallucinations.

LLM Claiming to be tuned for medical uses

with quadruple the context size of Clairity

U of Rhode Island

Needsa Grant (PI)

Cleveland State

Kent Loggin

Rochester Inst of Tech

Annette Work

U of CA Irvine

Ellie Vate

U of Washington

Lee King

NIH

Souper Yoozer

U Detroit Mercy

Arty Fishal

U of Alaska

Salmon Floyd

Clairity LLM Dev Team

OpenSkai LLM Dev Team

SkaiNet Investigator/AppDev Team

Research

Data Manager

UNC – Chapel Hill

Caroline Sweet

Real Diamond

SP1

SkaiNet

CILogon

IdP1

U RI

SP2

Clairity

NIAID

SP3

OpenSkai�Rice University

Day1

Day2

Day3

IdP2

U NC

IdP3

U Dt Mercy

IdP4 U Alaska

IdP6

cleveland OH

IdP5

U MO

IdP7

Rochester RIT

IdP8

U Wash

IdP9

U CA Irvine

IdP10

NIH

SP1

SkaiNet

Exercise Flow

ecc1

ecc2

Observations and Goals

• Many security teams not aware of Sirtfi or IAM paradigm
• Cultural barriers to sharing
• TLP barriers to sharing
• Thinking legally or reputationally vs operationally (risk of getting notification wrong can lead to under-sharing, slow response to securing systems)
• Continued need for participation – we can handle more!

​

​

• Every Jan: Call for Participation goes out for InCommon’s Sirtfi Exercise Planning Working Group (SEPWG)
• Every Summer a Call for Participation goes to InCommon and REFEDS mailing lists for a November TTX

Kyle Lewis, RDCT

Consider Playing Next Year

​

​

ACAMP Session for federation security?

Identity Assurance Deployment Guidance

| 20

REFEDS Assurance Framework – Quest complete, next quest begins…

We started a quest in 2021…

​

•2021-2023: updated REFEDS Assurance Framework (RAF)

•2023-2024: We wrote the risk assessment guide for US Govt SPs (and other SPs)

•2024-2025: We developed an implementation guide for InCommon IdPs

​

Deep dive into the Implementation Guide for IdPs today in this Plaza Room F at 1:40pm:

“Demystifying the REFEDS Assurance Framework”

​

Because now it’s time to do the thing.

​

​

| 21

Subject Identifiers �Access Entity Categories

Federation Proxies

…and more

| 22

Subject Identifiers defined

• * Defined in an OASIS spec in 2019
• https://docs.oasis-open.org/security/saml-subject-id-attr/v1.0/cs01/saml-subject-id-attr-v1.0-cs01.html
• Establishes two new identifiers
• subject-id: supercedes eppn
• pairwise-id: supercedes eduPersonTargetedID

| 23

Subject Identifier adoption

• * InCommon community WG wrapped up earlier this year
• Created guidance for the adoption of the new identifiers by deployers
• Not a quick task
• Want more? Stay in this room for the next session

| 24

Access Entity categories explained

• * Developed by SeamlessAccess and approved by REFEDS
• https://refeds.org/specifications
• Think R&S but better
• Defines anonymous, pseudonymous, and personalized access
• Adopted by InCommon, though not as widely used as it could be
• The new subject IDs are a key part of these categories

| 25

Federation Proxies

• * Proxies operating in the federation
• Could be an SP in front of other services
• Could be an IdP in front of other IdPs
• Could be a protocol bridge
• All scenarios need to address privacy and operating transparency
• Community WG wrapped up earlier this year
• http://doi.org/10.26869/TI.179.1.
• Next steps: put recommendations into practice

| 26

More good stuff from TAC

• * Efforts underway
• RP onboarding
• Device-level security
• Look for community working groups in 2026
• Longer-term joint efforts
• OpenID federation
• Federated authorization
• Will involve work from TAC, CTAB, and others
• Expect to hear more in 2026

​

| 27

| 28

REFEDS MFA Profile 2.0

| 29

REFEDS MFA Profile 2.0 – Primer

• Update to REFEDS MFA Profile 1.2
• request that MFA be used
• signal that MFA was used
• Support for phishing-resistant MFA
• backward compatible
• We must define phishing-resistant
• It must be practical and useful for RPs that want to be assured that MFA was used (e.g. NSF)

| 30

REFEDS MFA Profile 2.0 – Progress

• Being crafted carefully
• We have an opportunity to consider the problem space and use-case carefully and holistically
• Expected Q1-Q2 2026
• In the meantime, you should really be using MFA Profile 1.2 already
• or think about how to roll it out
• We want to hear your blockers and challenges!

| 31

Federation Expectations Program

| 32

What Do We Want?

Federation Practices for �Better Trusted Access!

| 33

What Do We Want?

Federation Practices for �Better* Trusted Access!

• Faster
• Greater Trust
• Fine-grained privileges
• Automation

• less manual customization
• bureaucratic overhead
• pre-provisioning
• protocol-specific

* “Better” might include

| 34

When Do We Want It ?

As Soon as We Agree�on Standards !

| 35

Precedents and Inspiration for Expectations Program

• Baseline Expectations
• Two successful rounds: contacts, TLS, Sirtfi, etc.
• Mandatory for ALL InC Federation entities
• So awkward for “advanced” needs with smaller scope�
• Deployment Guidelines (as described in this session!)
• Useful and flexible ways to introduce new capabilities
• Allow wide latitude in detail; are not “required”�
• Proposed Federation Expectations Program intended to address needs unmet by BE and DG

| 36

Federation Expectations Program

• Includes Baseline
• REQUIREMENTS for InCommon Federation
• Address fundamentals of trust and interoperability�
• Includes prescriptions - “how tos” for added value
• STANDARDS for features beyond “basic”
• “If this capability deployed, must conform to this standard”�
• Innovations are not in scope, but if and as they are “proven,” may become standards. Standards that address basic needs might eventually become Baseline Requirements.

| 37

5 Guiding Principles of Expectations Program� - Founded on the Success of Baseline

• This Community Leads and Decides�
• Expectations must strengthen trust, usability, scalability, and consistency�
• Expectations are practical, actionable, and achievable�
• Expectations recognize and support interoperability in local & global contexts �
• Federation Expectations is an Ongoing Program of Continuous Improvement

| 38

“Those are great high-minded goals -� Can We Coordinate to Make this Shift, Though ?”

Inspiration from Högertrafikomläggningen (Dagen H)��(but maybe we need a great logo and an official song)

| 39

First, we must walk

| 40

| 41

How it works

| 42

Lifecycle

Yearly

Federation Expectations

| 43

Lifecycle Rationale

• Repeatable
• Predictable
• Sustainable

Yearly

Federation Expectations

| 44

What does this mean for me?

| 45

What changes on day 1?

Nothing.

​

No new requirements.

​

This starts with conversation.

| 46

What you can expect

• More clarity
• Pathway to attain better practices
• Systematic community involvement
• Predictable timing
• Regular updates
• Improvements over the ad hoc ways we do things now

| 47

What we need from You

• Ideas
• Participation in the Conversation
• Consultation
• Surveys
• Working Groups
• Events
• Feedback

https://forms.gle/5BPkKFbnas9GK7Vt5

| 48

What’s Next?

| 49

Timeline*

* hopefully

• Q1 2026: Program setup continues
• Q2 2026: Organizational machinery up and running
• Q3–Q4 2026: First consultations

​

• Q1 2027: Incorporate feedback
• Q2 2027: Publish 2027 Edition
• Q3 2027: Revision and intake
• Q4 2027: Consultations
• Q1 2028: Incorporate feedback
• Q2 2028: Publish 2028 Edition
• …

| 50

Timeline*

* hopefully

• Q1 2026: Program setup continues
• Q2 2026: Organizational machinery up and running
• Q3–Q4 2026: Publish first high priority standards

​

• Q1 2027: Incorporate feedback
• Q2 2027: Publish 2027 Edition
• Q3 2027: Revision and intake
• Q4 2027: Consultations
• Q1 2028: Incorporate feedback
• Q2 2028: Publish 2028 Edition
• …

| 51

Stay Engaged

• Participate in Community Consultations
• Respond to Surveys
• Join the Conversation on Slack and Mailing Lists
• Attend Events
• InCommon Academy, Thread Meetups
• IAM Online
• Conferences: BaseCAMP, ACAMP
• Join a Working Group
• Want to hold an ACAMP session?

https://forms.gle/5BPkKFbnas9GK7Vt5

| 52

InCommon Federation Expectations gives us a way

to shape our own future together.

| 53