1 of 21

Confidence Interval

An AI-supported reasoning scaffold

for college students (and others)

Vaughn Tan

Singapore Government Centre for Strategic Futures

Winding Road Pte Ltd

UCL School of Management

2 of 21

An AI-powered tool

(a reasoning scaffold)

that helps users go from a thinly articulated argument like this

to a robustly conceptualised argument like this

in 15 minutes.

3 of 21

Reasoning scaffolds help humans reason explicitly and therefore more effectively about subjective things, not just objective truths.

Subjective reasoning is the most frequent cause of coordination problems —

yet remains a theoretically and practically under-explored area in epistemics.

Learn more about reasoning scaffolds here

4 of 21

Teaching undergrads through PhDs and exec MBAs in universities since 2008, and strategy consultant for govts/multilats/corps/startups since 2012.

tl;dr:

Ability to do subjective reasoning is declining.

Decline observed in students and non-students.

Leads to long-term coordination problems.

5 of 21

Boeing’s series of 737 MAX disasters show how coordination breaks down when different parts of an organisation unintentionally pursue different and conflicting subjective definitions of “good performance”: Increasing output volume, or compliance, or controlling production costs.

if you want evidence that doing this kind of human subj reasoning is important, take the recent Boeing 737 MAX disasters — two plane crashes killing 346 people and resulting in the global grounding of the 737 max fleet. i interpret this as an example of what happens when different parts of an organisation have different ideas of what is subjectively valuable as an outcome to pursue. some teams optimise for speed, others on compliance, and others on delivery schedules and cost control. Each team is doing what it believes is valuable — but the organisation as a whole is pursuing different subjectively valuable outcomes which conflict with each other. without effective reasoning as a whole organisation about what subjective outcomes to collectively value (for instance, maybe building safe, reliable aircraft) the organisation as a whole produced catastrophic failures. It’s a vivid example of coordination breaking down when what’s subjectively valued diverges across functions.

6 of 21

Decline accelerating with advent of AI tools.

Working on the problem for the last 3 years.

Testing mechanisms for enabling better human subjective reasoning since late 2024.

7 of 21

An early version of a reasoning scaffold.

8 of 21

Tested reasoning scaffolds using pen-and-paper mechanisms first, for

faster iteration and ability to modify scaffolds in real-time.

Real-world teams, real-world subjective reasoning about strategy for public utilities and economic development.

Very strong initial feedback:

Teams pirated test materials for own use, and participants requested more workshops using the test mechanism.

9 of 21

The Future of Life Foundation supported final testing and prototyping of a self-service, AI-supported version designed for college students.

10 of 21

I’ve made a quick video to demonstrate how a user runs through a session with Confidence Interval. These sessions are usually about 15 minutes long, but that’s still too long for a demo video so I’ve sped it up quite a bit. The logic of Confidence Interval is that the user builds the outline of a strong subjective argument by responding to prompts from the tool. You’ll see six key design principles operationalised in this prototype.

Explicitly scaffold reasoning steps. The process of constructing an argument is broken down and addressed step by step.
The LLMs working on the back end don’t generate new content. All they do is paraphrase what the user has entered. The AI systems that support Confidence interval function as a Socratic mirror only.
Confidence interval very clearly separates anything requiring subjective decisions or discretion from information retrieval — it keeps meaningmaking separate from other work involved in constructing a subjective argument.
And only the human user is permitted to do any of the meaningmaking work.
The goal in designing Confidence interval this way is to avoid replacing human instructors.
My intention is to free human instructors from the important but low-level and time-consuming tasks of providing reasoning scaffolds and doing socratic mirroring. Instead, the instructor can focus on higher-order coaching with the student, building on the output from Confidence Interval.

11 of 21

Design principles:

Explicitly scaffold reasoning steps.
Use AI systems as a Socratic mirror only.
Separate meaningmaking from other work.
Entrust meaningmaking only to human users.
Don’t try to replace human instructors.
Instead, free up instructor time for higher-order coaching on subjective reasoning.

Learn more about meaningmaking here

12 of 21

Feedback from student + instructor testers:

Massive instructor time savings (~ 60 min/stu).

Time savings for students (~45 min/stu).

Significant output quality improvements.

Applies to many types of subjective arguments.

13 of 21

“The 15 minutes I spent on this was more helpful than an entire year-long module on critical thinking.”

44 testers so far from educational institutions.

Over 95% want their institution to provide access.

14 of 21

Also being tested by government strategy groups, public utility operations teams, development agency leadership teams, consultancies, and startups.

15 of 21

“This is already helpful for clarifying our thinking. If we had a custom version for [our specific context] we would buy access for our middle management layer.”

16 of 21

Across the board, useful for improving arguments about subjective value judgments:

Justifying a business model

Deciding on an investment commitment

Choosing between product dev options

Arguing for a particular policy direction

17 of 21

Fully functioning prototype

ci1.vercel.app

18 of 21

Whitelist request form

forms.gle/Q8PYkEVFnoDaoggC6

19 of 21

Read the development note

vaughntan.org/aiux

20 of 21

Get in touch to learn more about the project.

I’d love your help with introductions to:

Institutions that want to test.

Funders who want to subsidise access.

People/teams who want to build it out with me.

21 of 21

21

^web

^email

^LinkedIn

^Farcaster

^X

^Bluesky