MATH-BIOINF-STATS 547: Mathematics of Data

​

Date: March 11, 2025

Dimension Reduction of Data: Examples

​

Cooper Stansbury

​

​

• Interesting data is high-dimensional, and often noisy�
• High-dimensional means each observation is described by a large number of variables or features
1. Motivating example: Single-cell gene expression data contains many cells, each with expression for several thousand genes X = (10000, 20000)�
• Discovering and communicating relationships in high-dimensional data can be tricky�
• Dimension reduction techniques simplify high-dimensional data while preserving critical information, making it easier to understand and communicate

High-Dimensional Data

• Each point is a single cell, colored by cell type, which is represented by a vector of 20000 genes
• Each group of points is a set of cells collected at a different time point
• Dimension reduction shows how populations change over time!

Data Example 1

Stansbury, Cooper M., Gabrielle A. Dotson, Harrison Pugh, Alnawaz Rehemtulla, Indika Rajapakse, and Lindsey A. Muir. "A lipid-associated macrophage lineage rewires the spatial landscape of adipose tissue in early obesity." JCI Insight. 2023 .

• Each point is a single cell, which is represented by a vector of 20000 genes
• Left: cells are clustered based on their gene expression
• Center: The expression of one gene (KLF2) in each cell
• Right: The expression of another gene (ETV6) in each cell
• Dimension reduction shows how groups of cells differ in expression of some genes

Data Example 2

C. Stansbury, J. Cwycyshyn, J. Pickard, W. Meixner, I. Rajapakse, and L. A. Muir, “Data-guided direct reprogramming of human fibroblasts into the hematopoietic lineage,” Aug. 26, 2024, bioRxiv

Principal Component Analysis (PCA)

• Linear method to summarize large datasets
• Identifies dimensions that explain the variability of the data�

t-Distributed Stochastic Neighbor Embedding (t-SNE)

• Non-linear dimensionality reduction technique that preserves local relationships
• Maps high-dimensional data to a lower-dimensional space by minimizing divergence between probability distributions�

Uniform Manifold Approximation and Projection (UMAP)

• Non-linear technique that balances global and local structure preservation
• Constructs a graph from high-dimensional data and optimizes a lower-dimensional embedding that preserves desired characteristics of this graph

​

Three Common Methods

Wang, Y., Huang, H., Rudin, C. and Shaposhnik, Y., 2021. Understanding how dimension reduction tools work: an empirical approach to deciphering t-SNE, UMAP, TriMAP, and PaCMAP for data visualization. The Journal of Machine Learning Research, 22(1), pp.9129-9201.

Overview

1. We are given a data matrix X, with size (observations x features)
2. Each algorithm takes X as input, and a few parameters that determine the final embedding
3. The output is a matrix D with size (observations, 2)
1. We reduce the dimension of the original data to 2 embedding vectors which can be easily visualized

PCA

1. Center the data
2. Calculate the covariance matrix
3. Find eigenvectors and eigenvalues
4. Project data onto top k=2 eigenvectors

t-SNE

1. Calculate pairwise similarities (high-D)
2. Calculate pairwise similarities (low-D, randomly initialized embeddings)
3. Minimize the difference between the two similarity distributions (gradient descent with KL-divergence as the cost function)

UMAP

1. Build a weighted k-nearest neighbor graph (high-D)
2. Optimize a low-dimensional embedding to match the graph (initialized randomly, or from the spectral embedding of the symmetric Laplacian)

Overview of PCA, t-SNE, and UMAP

PCA: From Cleve

PCA without SVD

X = A - mean(A)

C = (X'*X)/(n-1)

[Q,E] = eig(C)

coeff = Q

latent = diag(E)

score = X*coeff

​

PCA from SVD

X = A - mean(A)

[U,S,V] = svd(X,'econ')

coeff = V

latent = diag(S).^2/(n-1)

score = U*S

​

A Simple Dataset (iris)

• The information in the previous slide is summarized in two dimensions
• All three methods capture that setosa is the most different, and that veriscolor and viriginica are more similar - but do so in a single plot!
• Python code is here: https://github.com/CooperStansbury/dimension_reduction

A Simple Dataset (iris)

• SVD/PCA are amazing - you should always try this first!�
• The parameters chosen for UMAP/t-SNE impact the embedding
• experimentation is key - try different values
• https://projector.tensorflow.org/
• https://pair-code.github.io/understanding-umap/�
• UMAP/t-SNE are primarily for visualizing data - but SVD/PCA embeddings can be used for downstream analysis
• For example, distances between t-SNE embeddings are not good measures of similarity - but distances between PCA embeddings might be�
• CPU-bound t-SNE can be quite slow on large datasets - UMAP is faster�
• Practical considerations:
• Filtering data is always a good idea
• Understand the rank of your data
• For single-cell RNA-seq, combining PCA + UMAP usually leads to nice embeddings!
• GPU-based implementations of t-SNE/UMAP are available (RAPIDS)

A few notes and practical considerations

Supplementary Material

Van der Maaten, Laurens, and Geoffrey Hinton. "Visualizing data using t-SNE." Journal of machine learning research 9.11 (2008).

The Original Algorithm for t-SNE

Algorithm for PCA

Algorithm for t-SNE

Algorithm for UMAP

• Python:
• scikit learn:
• https://scikit-learn.org/stable/api/sklearn.manifold.html
• https://scikit-learn.org/stable/api/sklearn.decomposition.html
• https://scikit-learn.org/stable/modules/decomposition.html
• UMAP

​

• MATLAB:
• https://lvdmaaten.github.io/drtoolbox/#download�
• Interactive tools:
• https://projector.tensorflow.org/
• https://pair-code.github.io/understanding-umap/�

Additional Resources

Example 2

C. Stansbury, J. Cwycyshyn, J. Pickard, W. Meixner, I. Rajapakse, and L. A. Muir, “Data-guided direct reprogramming of human fibroblasts into the hematopoietic lineage,” Aug. 26, 2024, bioRxiv

Clustering