Big data analysis

Probability

Covid

nonCovid

positive

negative

positive

negative

0.001

0.999

0.005

0.995

0.99

0.01

0.001*0.99/(0.001*0.99+0.999*0.005)=16.54%

Disease effect

Covid

nonCovid

positive

negative

positive

negative

0.5

0.5

0.005

0.995

0.99

0.01

0.5*0.99/(0.5*0.99+0.5*0.005)=99.5%

Test accuracy effect

Covid

nonCovid

positive

negative

positive

negative

0.5

0.5

0.005

0.995

0.99

0.01

0.001*0.99/(0.001*0.999+0.999*0.001)=50.02%

Radiant skills

• Visualize to investigate correlation in data
• Transform -> Create : make new variables

ifelse(condition, yes, no) if condition, yes, else no

sub(A, B, C) A->B otherwise, C

• Linear regression : P value, R2 value, correlations(model->estimate)
• Validate : training=1or0

Validation without test data: Linearity

• Assumptions about the population:
• Y_i = β₀+ β₁x_1i+ … + β_kx_ki + Ɛ_i
• Ɛ₁ , Ɛ₂ ,…, Ɛ_n are i.i.d. random variables, N(0, σ)
• Linearity
• If k=1 (simple regression), one can check visually using a scatter plot.
• “Sanity check”: the sign of the coefficients, reason for non-linearity

Validation: Normality

• Normality of Ɛ_i
• Plot a histogram of the residuals
• Check “Normal Q-Q” in “Dashboard”
• Should be an (almost) diagonal line
• Fortunately, conclusions are usually fairly robust to deviations from this assumption

Validation: Homoscedasticity (constant variance)

• Do the error terms have constant standard deviation? �(i.e. SD(Ɛ_i) = σ for all i)
• Check scatter plots of residuals vs. all variables.
• Example: Ideal

Validation: Heteroscedasticity (non-constant variance)

Validation: Heteroscedasticity

-20.00

-10.00

0.00

10.00

20.00

0.0

1.0

2.0

No evidence of heteroscedasticity

Evidence of heteroscedasticity

Residuals

Heteroscedasticity: possible remedies

• Transformation of the response variable

Validation: Autocorrelation (independence)

• Are the error terms independent?
• Autocorrelation may be present if the observations have a natural sequential order
• Plot the residuals in row order and check for patterns

No evidence of autocorrelation

Evidence of autocorrelation

Validation: Check the ‘Dashboard’

Interpretation

Opportunity cost

Ln(10000)=9.21

9.21-7.86=1.35=1.882*ln(Carat)+Quality

For example

Logistic regression (model->esitmate)

How to get big data?

-Use public data (from government agents)

-Collect using sensors

-Scrape from website

google analytics, amazon

Google analytics

Scrape

Reviews: 9505 in Amazon

Review analysis

Rocco & Roxie (analysis of data from only Amazon)

32oz

1 gallon

Profit estimation

Rocco

Angry Orange

Zero Odor

Febreze

Brand trust value from cross-purchasing (rocco)

Stain eliminator: 9505 reviews

Jerky: 3372 reviews

Supplement: 318 reviews

Shampoo: 313 reviews

Shared reviewers

Stain eliminator & Jerky: 830 reviewers

Stain eliminator & Supplement: 126 reviewers

Stain eliminator & Shampoo: 127 reviewers

https://www.amazon.com/Chamomile-Aromatherapy-Freshening-Pet-Eliminator/dp/B00J3XGFZO/ref=sr_1_5?keywords=dog+perfume&qid=1568847386&s=gateway&sr=8-5

Ingredients

Purified Water, Vegetable-derived Glycerin, Natural Coconut-derived Cleansers, Vegetable-derived Quaternary Ammonium Salt, Isopropyl Alcohol, Castor Bean Oil, Natural Vegetable-derived Hydrotope, Fragrance, Aloe Vera Extract, Lanolin Lanolin-based Conditioner, Lavender Oil, Lanolin, Paraben Free Preservative, Pro-Vitamin B5, Oat Extract, Lavender Oil�Directions

Avoiding eyes, generously spray onto wet or dry coat. Comb or brush through coat to evenly distribute and clean. Finish with an all over mist to perfume.

Gerrard Larriett