R-Radiant

heights.csv

Visualize->scatter

Create a new variable to denote who is taller in a mother-daughter pair

Change its type to factor

Create another variable to break data into 3 tiers depending on mother height

Change its type to factor

Need to reorder levels to something more natural

Reorder levels

Much better

Interpret!

Simple model = linear regression model

f(x1,x2,x3,x4,…..)=y

Sales [M$] By State

Nature.rda

Total Deviation: Total Sum of Squares (SST)

Sales [M$] by advertising expenditures & state

A linear relationship?

A linear relationship?

ŷ = b₀ + b₁x

Unexplained deviation (“errors”)

Expected sales at $0.8M Advertising?

ŷ = b₀ + b₁x0.8

Predicted value

Total deviation = explained deviation + error

Total

deviation

Unexplained deviation (“errors” or “residuals”)

Unexplained deviation: Sum of Squared Errors (SSE)

Unexplained deviation: Sum of Squared Errors (SSE)

Regression (OLS) determines the line that minimizes the

Sum of Squared Errors. i.e., b₀ and b₁ are determined such that

they minimize:

​

​

Note: Also known as Residual Sum of Squares

Explained deviation

Explained deviation: Sum of Squares Regression(SSR)

Adding up deviations

​

​

​

​

SSR

​

Sum of Squares

Regression

​

​

​

​

SSE

​

Sum of Squared Errors

​

​

​

​

SST

​

Total

Sum of Squares

​

=

+

+

=

Simple Linear Regression – The Basic Model

• Observed data: (x₁, y₁), (x₂, y₂),…, (x_n, y_n)
• x₁, x₂,…, x_n are n given values of the independent variable
• y₁, y₂,…, y_n are the corresponding n observations of the dependent variable

​

• Model of the population: Y_i = β₀+ β₁x_i+ Ɛ_i
• Ɛ₁ , Ɛ₂ ,…, Ɛ_n are i.i.d. random variables, N(0, σ)
• This is the true relation between Y and x, yet we do not know β₀ and β₁ and have to estimate them based on the observed data.

Population Model : Y_i = β₀+ β₁x_i+ Ɛ_i

ε_i~N(0,σ)

β₁

β₀

Comments

​

​

• Relationship is linear – described by a “line”
• β₀ = “baseline” value of Y
• (i.e. value of Y if x is 0)
• β₁ = “slope” of line
• (average change in Y per unit change in x)

E(Y_i|x_i) =

SD(Y_i|x_i) = σ

β₀+ β₁x_i

Notation

• Regression coefficients: b₀ and b₁ are estimates of β₀ and β₁

​

• Regression estimate for Y at x_i (prediction): ŷ_i = b₀ + b₁x_i

​

• Residuals: e_i = y_i - ŷ_i

​

Realization: y_i = b₀+ b₁x_i+ e_i

y₅

e₅

b₁

b₀

Multiple Linear Regression

• Observed data:
• (x₁₁, x₂₁,…, x_k1), (x₁₂, x₂₂,…, x_k2),…, (x_1n, x_2n,…, x_kn) are records of given values of the explanatory variables
• y₁, y₂,…, y_n are the n observations of the dependent variable

​

• Population model:
• Y_i = β₀+ β₁x_1i+ … + β_kx_ki + Ɛ_i
• Ɛ₁ , Ɛ₂ , … Ɛ_n are i.i.d. random variables, N(0, σ)

Multiple Linear Regression

• Regression coefficients: b₀,b₁,…, b_k are estimates of β₀ , β₁,..., β_k
• Prediction for Y at x_i (prediction): �ŷ_i = b₀ + b₁x_1i + … + b_kx_ki
• Residual (error): e_i = y_i - ŷ_i
• Goal: choose b₀, b₁,…, b_k to minimize the sum of squared error

Estimating a regression model

in Radiant

Nature.rda

Plot or perish!

Do the ‘signs’ make sense?

Regression output

Regression output: Coefficients

b₀, b₁,…, b_k are estimates of the true parameters β₀, β₁,…, β_k

Interpreting regression coefficients

• Regression coefficients: b₀,b₁,…, b_k are estimates of β₀ , β₁ ,..., β_k Fact: E(b_i) = β_i

​

• Example:
• b₀ = 65.70 (Intersection with Y-axis. Interpret with care or ignore)
• b₁= 48.98 ($1 million increase in advertising is expected to increase sales by $49 million, keeping all else constant)
• b₂= 59.65 ($1 million increase in promotions is expected to increase sales by $59.7 million, keeping all else constant)
• b₃= -1.84 ($1 million increase in competitor sales is expected to decrease sales by $1.8 million, keeping all else constant)

Regression output: Standard Error

σ

Regression output: Standard Error

• Standard error s: an estimate of σ, the standard deviation of the Ɛ_i. It is a measure of the spread of the data around the line and represents the amount of “noise” in the model.
• Example: s = 17.60

Regression output:

Sum of Squares

SST: “Total Sum of Squares”

SSE: “Sum of Squared Errors”

SSR: “Sum of Squares Regression”

Regression output:

R²

Coefficient of determination(R²)

​

A measure of the overall strength of the relationship between the Response and Predictor variables

​

How much of the variation has been explained?

Understanding regression output

Coefficient of determination: R²

• R² = 1; x values account for all of the variation in the Y values

• R² = 0; x values account for none of the variation in the Y values

Coefficient of determination: R²

• R² = 0.8; x values account for most of the variation in the Y values

• R² = 0.4; x values account for some of the variation in the Y values

Standard deviations for the coefficients

T values for the coefficients

“distance” of the coefficients from 0, measured in standard deviations. i.e., t.value = coeff./std.error

P values for the coefficients

p.value is the probability of finding a t.value of this size if the null-hypothesis is true

95% confidence interval for the coefficients.

If the interval does not contain 0, our confidence that the variable has explanatory power is (at least) 95%.

Interpreting confidence intervals

If the interval DOES NOT contain 0 we conclude that β_i is significantly different from zero

​

🗴 Bad :

✓ Good :

Interpreting confidence intervals

Validation without test data: Linearity

• Assumptions about the population:
• Y_i = β₀+ β₁x_1i+ … + β_kx_ki + Ɛ_i
• Ɛ₁ , Ɛ₂ ,…, Ɛ_n are i.i.d. random variables, N(0, σ)
• Linearity
• If k=1 (simple regression), one can check visually using a scatter plot.
• “Sanity check”: the sign of the coefficients, reason for non-linearity

Validation: Normality

• Normality of Ɛ_i
• Plot a histogram of the residuals
• Check “Normal Q-Q” in “Dashboard”
• Should be an (almost) diagonal line
• Fortunately, conclusions are usually fairly robust to deviations from this assumption

Validation: Homoscedasticity (constant variance)

• Do the error terms have constant standard deviation? �(i.e. SD(Ɛ_i) = σ for all i)
• Check scatter plots of residuals vs. all variables.
• Example: Ideal

Validation: Heteroscedasticity (non-constant variance)

Validation: Heteroscedasticity

-20.00

-10.00

0.00

10.00

20.00

0.0

1.0

2.0

No evidence of heteroscedasticity

Evidence of heteroscedasticity

Residuals

Heteroscedasticity: possible remedies

• Transformation of the response variable

Validation: Autocorrelation (independence)

• Are the error terms independent?
• Autocorrelation may be present if the observations have a natural sequential order
• Plot the residuals in row order and check for patterns

No evidence of autocorrelation

Evidence of autocorrelation

Validation: Check the ‘Dashboard’

Let’s practice diamond!!

Split data for validation

Training data for building model

Test data for validation of the model

-Create new vaiable for training/test

-For training, give 1.

-For test, give 0.

Data->trasform->trasformation type-> training variable

70% of data will have 1 in ‘training’ variable

Random function

This enables various selections

2100 data out of 3000 data is ’training=1’

data -> view -> filter -> training ==1 -> store in ‘diamond_training’

data -> view -> filter -> training ==0 -> store in ‘diamond_test’

Select ‘diamond_training’ and build linear regression

65% probability of x is not contributing on price

Let’s look at the data and variables.

• carat = weight of the diamond (0.2–3.00)
• depth = total depth percentage = z / mean(x, y) = 2 * z / (x + y) (54.2–70.80)
• table = width of top of diamond relative to widest point (50–69)
• x = length in mm (3.73–9.42)
• y = width in mm (3.71–9.29)
• z = depth in mm (2.33–5.58)

These are all talking about ‘size’

x

y

z

table

• depth = total depth percentage = z / mean(x, y) = 2 * z / (x + y) (54.2–70.80)

X,y,z are stronlgy correlated, and that is why depth is less corelated.

D,E->A

Change type!! (as a factor)

Just in case, make a variable for color D

Color F+D(yes)??

Something wrong?

Carat vs color?

Depth, table evaluation

Exclude table, and play with “size” parameters

Can you predict price of x>y diamond (various depth) using your model?

Finalize the model and run the prediction!

Prediction->data->choose test data->run and save the data

Don’t forget to create the variable of color_new in test data to run the prediction!!

D,E->A

Change type!! (as a factor)

Additional analysis! Look at the data again

All data

Pred<0 data

Smaller carat

(smaller x,y,z)

Higher depth

(depth = z / mean(x, y) = 2 * z / (x + y))

​

Segmentation! Let’s build a new linear regression for carat <0.5.

How many segmentation you can take?

data

Carat>0.5

Carat<0.5

Color=D,E

Other color

Other color

Color=D,E

Risiduals = Price – Predicted Price

Test data

Two approaches

1. Segmentation-split data into two (carat<1, 1<carat) and build two linear regression
2. Reflect to expand the data of carat<1, HOW?

​

​

+Run nueral network and ask it do something for us

Too dispersed

Too condensed

The market looks like it prefers 1, 1.5, 2 carats diamond

How we can make data even distributed?

Take log or ln : log makes larger number into smaller

Test data

What happened to ”color”?

Negative

Positive?

​

lnPrice=1.882*lnCarat+color+cut+clarity+7.855

​

With $10,000

Max Carat?

ln(10,000)=1.882*lnCarat-0.5+7.855

Carat=2.68, clarity=I1, cut=fair, color=J

​

Best quality?

ln(10,000)=1.882*lnCarat+1.089+0.168+7.855

Carat=1.05, clarity=IF, cut=Ideal, color=D

Color

New variable

Neural Network Method

Color

Model

f(x1,x2,x3,..)=a

a*f(x1,x2,x3,..)=price

Require the setup to shorten the calculation amount