1 of 52

The Scalable Analytics Institute (ScAI)�Department of Computer Science�University of California, Los Angeles (UCLA)

Instructor: Jeehyun Hwang

Lecture 1: Classification

Data Science for All 2023

Data Science for All 2023

Lecture 1: Classification

2 of 52

It’s OK that you do not know many things

2

3 of 52

Don’t lose your big picture

3

https://en.wikipedia.org/wiki/Fractal

4 of 52

What is Classification?

4

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

5 of 52

Supervised Learning: Regression

  • Regression: the label to be predicted is continuous
    • Example: predict a new house price based on its living area, street, sale condition, …

5

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

6 of 52

Supervised Learning: Classification

  • Classification: the label to be predicted is discrete
    • Example: given an image, predict its semantic category

6

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

7 of 52

Overview of Linear Regression

7

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

8 of 52

Regression Data: Advertising Dataset

8

July 17, 2023

TV

radio

newspaper

sales

230.1

37.8

69.2

22.1

44.5

39.3

45.1

10.4

17.2

45.9

69.3

9.3

151.5

41.3

58.5

18.5

180.8

10.8

58.4

12.9

Y

outcome

response variable

dependent variable

​

X

predictors

features

covariates

​

p predictors

n observations

Data Science for All 2023

Lecture 1: Classification

9 of 52

Linear Models

9

July 17, 2023

 

 

Data Science for All 2023

Lecture 1: Classification

10 of 52

Linear Regression

10

July 17, 2023

 

Our Linear Model!

Data Science for All 2023

Lecture 1: Classification

11 of 52

Linear Regression: Example

11

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

12 of 52

Estimation of Regression Coefficients

12

July 17, 2023

For a given data set

Data Science for All 2023

Lecture 1: Classification

13 of 52

Estimation of Regression Coefficients (Cont.)

13

July 17, 2023

For a given data set

Data Science for All 2023

Lecture 1: Classification

14 of 52

Estimation of Regression Coefficients (Cont.)

14

July 17, 2023

Maybe this one?

Data Science for All 2023

Lecture 1: Classification

15 of 52

Estimation of Regression Coefficients (Cont.)

15

July 17, 2023

Or this one?

Data Science for All 2023

Lecture 1: Classification

16 of 52

Question

  • Which line is better?
  • How to measure the goodness of a line?

​

16

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

17 of 52

Estimation of Regression Coefficients (Cont.)

17

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

18 of 52

Estimation of Regression Coefficients (Cont.)

  •  

18

July 17, 2023

 

 

 

Data Science for All 2023

Lecture 1: Classification

19 of 52

Exercise: Residual and Loss Calculation

19

July 17, 2023

 

​

​

Residual/error

1

7

​

2

14

​

4

21

​

Data Science for All 2023

Lecture 1: Classification

20 of 52

Back to Classification

20

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

21 of 52

Classification: Heart Data

21

July 17, 2023

Age

Sex

ChestPain

RestBP

Chol

Fbs

RestECG

MaxHR

ExAng

Oldpeak

Slope

Ca

Thal

AHD

63

1

typical

145

233

1

2

150

0

2.3

3

0.0

fixed

No

67

1

asymptomatic

160

286

0

2

108

1

1.5

2

3.0

normal

Yes

67

1

asymptomatic

120

229

0

2

129

1

2.6

2

2.0

reversable

Yes

37

1

nonanginal

130

250

0

0

187

0

3.5

3

0.0

normal

No

41

0

nontypical

130

204

0

2

172

0

1.4

1

0.0

normal

No

​

response variable Y

is Yes/No

Data Science for All 2023

Lecture 0: Overview

22 of 52

Heart Data

  • These data contain a binary outcome HD for 303 patients who presented with chest pain. An outcome value of:
  • Yes indicates the presence of heart disease based on an angiographic test,
  • No means no heart disease.
  • There are 13 predictors including:
  • Age
  • Chol (a cholesterol measurement),
  • MaxHR
  • RestBP
  • and other heart and lung function measurements.

​

22

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

23 of 52

Classification

  • When the response variable is categorical, then the problem is no longer called a regression problem but is instead labeled as a classification problem.
  • The goal is to attempt to classify each observation into a category (aka, class) defined by Y, based on a set of predictor variables X.

23

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

24 of 52

Classification

  • Up to this point, the methods we have seen have centered around modeling and the prediction of a quantitative response variable. Linear regression perform well under these situations
  • When the response variable is categorical, then the problem is no longer called a regression problem but is instead labeled as a classification problem.
  • The goal is to attempt to classify each observation into a category (aka, class) defined by Y, based on a set of predictor variables X.

24

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

25 of 52

Typical Classification Examples

  • The motivating examples for this lecture(s) are based [mostly] on medical data sets. Classification problems are common in this domain:
  • Trying to determine where to set the cut-off for some diagnostic test (pregnancy tests, prostate or breast cancer screening tests, etc...)
  • Trying to determine if cancer has gone into remission based on treatment and various other indicators
  • Trying to classify patients into types or classes of disease based on various genomic markers

25

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

26 of 52

A Classification Example

26

July 17, 2023

 

Linear regression does not work well, or is not appropriate at all, in this setting.

Data Science for All 2023

Lecture 1: Classification

27 of 52

A Classification Example (Cont.)

27

July 17, 2023

  • A linear regression could be used to predict y from x. What would be wrong with such a model?
  • The model would imply a specific ordering of the outcome, and would treat a one-unit change in y equivalent. The jump from y = 1 to y = 2 (CS to Statistics) should not be interpreted as the same as a jump from y = 2 to y = 3 (Statistics to everyone else).
  • Similarly, the response variable could be reordered such that y = 1 represents Statistics and y = 2 represents CS, and then the model estimates and predictions would be fundamentally different.
  • If the categorical response variable was ordinal (had a natural ordering, like class year, Freshman, Sophomore, etc.), then a linear regression model would make some sense but is still not ideal.

Data Science for All 2023

Lecture 1: Classification

28 of 52

Even Simpler Classification Problem: Binary Response

28

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

29 of 52

Probabilistic interpretation

29

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

30 of 52

Probability?

30

July 17, 2023

  • What is your most frequently used social media?

Data Science for All 2023

Lecture 1: Classification

31 of 52

Probability? (Cont.)

  • If you are from abroad (non-U.S. countries), answer the same question again. What is your most frequently used social media?

​

31

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

32 of 52

Probability? (Cont.)

  • If you are from U.S., answer the same question again. What is your most frequently used social media?

​

32

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

33 of 52

What have we learned just now?

  • Probability
    • P(Y)

​

  • Conditional probability
    • P(Y|X=“abroad”)
    • P(Y|X=“U.S.”)

​

  • Yes, we don’t need a stats course to understand this

​

33

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

34 of 52

Even Simpler Classification Problem: Binary Response (cont)

34

July 17, 2023

  • What could go wrong with this linear regression model?

​

Data Science for All 2023

Lecture 1: Classification

35 of 52

Even Simpler Classification Problem: Binary Response (cont)

35

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

36 of 52

36

Logistic Regression

Data Science for All 2023

Lecture 1: Classification

37 of 52

Logistic Function

37

July 17, 2023

 

Think of a function that would do this for us

Data Science for All 2023

Lecture 1: Classification

38 of 52

Logistic Function

38

July 17, 2023

 

 

Data Science for All 2023

Lecture 1: Classification

39 of 52

Logistic Regression

39

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

40 of 52

 

40

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

41 of 52

 

41

July 17, 2023

Data Science for All 2023

Lecture 1: Classification

42 of 52

Logistic Regression

42

July 17, 2023

 

Data Science for All 2023

Lecture 1: Classification

43 of 52

Estimation in Logistic Regression

43

July 17, 2023

    • In linear regression what loss function was used to determine the parameter estimates?
    • Which loss function we should use for logistic regression?

​

​

​

​

Data Science for All 2023

Lecture 1: Classification

44 of 52

Which Logistic Curve is Better?

44

July 17, 2023

  • This one?

​

​

​

Data Science for All 2023

Lecture 1: Classification

45 of 52

Which Logistic Curve is Better?

45

July 17, 2023

  • This one?

​

​

​

Data Science for All 2023

Lecture 1: Classification

46 of 52

Which Logistic Curve is Better?

46

July 17, 2023

  • This one?

​

​

​

Data Science for All 2023

Lecture 1: Classification

47 of 52

Check this toy example: compute probabilities

47

July 17, 2023

 

Note:

Data Science for All 2023

Lecture 1: Classification

48 of 52

Check this toy example: compute probabilities

48

July 17, 2023

 

Note:

Data Science for All 2023

Lecture 1: Classification

49 of 52

Check this toy example: compute probabilities

49

July 17, 2023

 

Note:

Data Science for All 2023

Lecture 1: Classification

50 of 52

Which prediction is better?

50

July 17, 2023

​

​

Observed Y

Predicted P for Y=1

​

​

​

​

​

​

​

​

Observed Y

Predicted P for Y=1

​

​

​

​

​

​

​

​

 

Data Science for All 2023

Lecture 1: Classification

51 of 52

The logic behind estimation

51

July 17, 2023

​

​

 

Data Science for All 2023

Lecture 1: Classification

52 of 52

52

July 17, 2023

Thank You

Data Science for All 2023

Lecture 1: Classification