1 of 73

Dimensionality Reduction with Principal Component Analysis

André E. Lazzaretti

Universidade Tecnológica Federal do Paraná (UTFPR) - Curitiba

Pós-Graduação em Engenharia Elétrica e Informática Industrial (CPGEI)

2 of 73

Introduction

High-dimensional data is often overcomplete, i.e., many dimensions are redundant and can be explained by a combination of other dimensions.
Dimensions in high-dimensional data are often correlated so that the data possesses an intrinsic lower-dimensional structure.

3 of 73

Problem Setting

4 of 73

Problem Setting

5 of 73

Problem Setting

6 of 73

Problem Setting

7 of 73

Problem Setting

8 of 73

Problem Setting

9 of 73

Problem Setting

10 of 73

MNIST Dataset

http://yann.lecun.com/exdb/mnist/

11 of 73

Maximum Variance Perspective

Our aim is to find a matrix B that retains as much information as possible when compressing data by projecting it onto the subspace spanned by the its columns.
Retaining most information after data compression is equivalent to capturing the largest amount of variance in the low-dimensional code.

12 of 73

Maximum Variance Perspective

13 of 73

Direction with Maximal Variance

14 of 73

Direction with Maximal Variance

15 of 73

Direction with Maximal Variance

16 of 73

Direction with Maximal Variance

17 of 73

Direction with Maximal Variance

18 of 73

Direction with Maximal Variance

19 of 73

Direction with Maximal Variance

20 of 73

Direction with Maximal Variance

21 of 73

Direction with Maximal Variance

22 of 73

Direction with Maximal Variance

23 of 73

Direction with Maximal Variance

24 of 73

M-dimensional Subspace with Maximal Variance

mth principal component can be found by subtracting the effect of the first m-1 principal components from the data:

25 of 73

M-dimensional Subspace with Maximal Variance

26 of 73

M-dimensional Subspace with Maximal Variance

27 of 73

M-dimensional Subspace with Maximal Variance

28 of 73

M-dimensional Subspace with Maximal Variance

29 of 73

M-dimensional Subspace with Maximal Variance

30 of 73

M-dimensional Subspace with Maximal Variance

eigenvector that is not among the first m-1 principal components

b_i is a basis vector of the principal subspace onto which B_m−1 projects.

31 of 73

M-dimensional Subspace with Maximal Variance

32 of 73

M-dimensional Subspace with Maximal Variance

33 of 73

M-dimensional Subspace with Maximal Variance

34 of 73

M-dimensional Subspace with Maximal Variance

35 of 73

Projection Perspective

36 of 73

Setting and Objective

37 of 73

Setting and Objective

38 of 73

Setting and Objective

39 of 73

Setting and Objective

40 of 73

Finding Optimal Coordinates

41 of 73

Finding Optimal Coordinates

42 of 73

Finding Optimal Coordinates

43 of 73

Finding Optimal Coordinates

44 of 73

Finding Optimal Coordinates

45 of 73

Finding Optimal Coordinates

46 of 73

Finding Optimal Coordinates

47 of 73

Finding Optimal Coordinates

48 of 73

Finding Optimal Coordinates

49 of 73

Finding Optimal Coordinates

50 of 73

Finding Optimal Coordinates

51 of 73