1 of 16

Rayid Ghani, Conrad Tucker

How do we build an AI system?

Designing Better Human-AI Futures

Designing Better Human-AI Futures

2 of 16

Designing Better Human- AI Futures

How do we design an AI system to achieve those values?

What values should an AI system have?

How do we validate that it leads to those societal values?

Intro to AI

Intro to Design Thinking

Deeper Dive into Dealing with Bias and Equity Issues in AI

Designing Better Human-AI Futures

3 of 16

Designing Better Human- AI Futures

How do we design an AI system to achieve those values?

What values should an AI system have?

How do we validate that it leads to those societal values?

Intro to AI

Intro to Design Thinking

Deeper Dive into Dealing with Bias and Equity Issues in AI

Designing Better Human-AI Futures

4 of 16

A simple definition of Machine Learning

“A computer program is said to learn from experience E with respect to some task T and some performance measure P, if its performance on T, as measured by P, improves with experience E.”

​

Designing Better Human-AI Futures

5 of 16

For your project:

Task T:

​

Experience E:

​

Performance Measure P:

Designing Better Human-AI Futures

6 of 16

Typical AI Capabilities

  • Description (Understand the past)
  • Detection (Anomalies, Events, Patterns)
  • Prediction (Predict the Future)
  • Optimization (Decision-Making)
  • Behavior Change (Causal Inference)

Designing Better Human-AI Futures

7 of 16

Process

Define Problem: Identify Goals, Values, and Desired Outcomes

Data Acquisition:

Identify existing data and/or augment with new data

Design AI System:

Test different approaches, compare, and select the one that best matches desired goals and values

​

Evaluate AI System:

Run trials and measure outcomes

​

Monitor and Adapt:

Alert if things change and adjust the AI system

​

Designing Better Human-AI Futures

8 of 16

How do we train an AI system?

Designing Better Human-AI Futures

9 of 16

Supervised machine learning framework

= f( )

output

“Learned”

function

Features/Predictors

y

X

Designing Better Human-AI Futures

10 of 16

How to solve a prediction problem

  • Define and Create Label (outcome variable)
  • Define and Create Features (predictors)
  • Create Training and Validation Sets
  • Train model(s) on Training Set
  • Validate model(s) on Validation Set
  • Select “best” model

​

Designing Better Human-AI Futures

11 of 16

Model Selection

  • You’ve tried and built a large number of different types of models

​

Now:

  • You need to understand what types of models work when, and
  • You need to decide which one(s) to use in the future

Designing Better Human-AI Futures

12 of 16

What we need to validate

  • Methodology
  • Metric(s)
  • Comparing to baselines

Designing Better Human-AI Futures

13 of 16

Evaluation - Metrics

  • Predictions are often scores between 0 and 1
  • We need to first turn them into 0 or 1 by selecting a threshold

​

​

Yes

No

Yes

True Positives (TP)

False Negatives (FN)

No

False Positives (FP)

True Negatives (TN)

Predicted

Actual

Designing Better Human-AI Futures

14 of 16

Evaluation – Metrics (at a threshold k)

  • Accuracy = (TP + TN) / (TP + TN + FP + FN)

​

  • Precision (or PPV) = TP / (TP + FP)�
  • Recall (or Sensitivity) = TP / (TP + FN)�
  • Specificity = TNR

​

​

Yes

No

Yes

True Positives (TP)

False Negatives (FN)

No

False Positives (FP)

True Negatives (TN)

Predicted

Actual

Designing Better Human-AI Futures

15 of 16

Varying the Threshold

Designing Better Human-AI Futures

16 of 16

Reminders

  • Assignment Due March 1
  • Weekly feedback

Designing Better Human-AI Futures