1 of 45

General(ized) Linear Models: an introduction to machine learning

Lior Pachter

California Institute of Technology

Lecture 11

Caltech Bi/BE/CS183

Spring 2023

These slides are distributed under the CC BY 4.0 license

2 of 45

Map of distributions (Lecture 10)

The distributions we have discussed are highlighted in red.
“Pascal distribution” is another name for the negative binomial distribution.
These are relevant for basic analysis of count data. There are many other distributions that are useful in computational biology.�
We will examine how simple linear regression can be extended to model dependent variables and noise that do not have Gaussian distributions.

Leemis et al.

3 of 45

Simple linear regression

We previously discussed linear regression (Lecture 3).
Our focus was on simple linear regression which is solved by least squares. That is, the model for pairs of points (x_i,y_i) was�

4 of 45

Multiple (linear) regression

Multiple regression refers to a regression in which there is one dependent variable and multiple independent variables:��
The same linear algebra approach that is used to solve simple linear regression works for multiple regression as well.

5 of 45

Recall (Lecture 3): solving the least squares problem

Method 2: Linear algebra.�
Consider a column vector formed from the dependent variables, i.e. Y = (y₁,...,y_n)^T, as a point in an n-dimensional vector space. Observe that the least squares optimization problem is equivalent to finding the nearest point on a subspace spanned by a column matrix X defined from the dependent variables. Formally, we seek to find the value β that minimizes (||Xβ - Y||₂)²; the minimal β is denoted β.�
Using orthogonality, the solution emerges naturally as (X^TX)^-1X^TY. This elegant and simple formula is true Babylonian bliss.

standard notation for an estimator

6 of 45

General linear model

The general linear model is the set of models��
The parameters for each dependent variable y_{. j} are estimated independently, so regression with the general linear model, i.e. general multivariate regression, is equivalent to a sequence of independent multiple linear regressions.

. . .

7 of 45

Regression types

Simple linear regression

Multiple linear regression

General linear models

Dependent variables

Independent variables

outcomes

8 of 45

Generalized linear models

Generalized linear models (GLMs) can be thought of as machines that generate data:�

The machine is equipped with

A fixed set of parameters called β.
A family of distributions from a class called the exponential family.
A function g called the link function.

To generate an observation from data X:

Compute Xβ.
Set μ = g^-1(Xβ).
Using the distribution with mean μ from the family chosen, generate an observation Y.

9 of 45

Generalized linear models

Formally, GLMs consist of three parts:

An exponential family of probability distributions.
A linear predictor function.
A link function g where μ := E[ Y | X ] = g^-1(Xβ).�

In simple linear regression:

The exponential family is the family of normal distributions.
The link function is the identity function, i.e. �E[ Y | X ] = Xβ.

10 of 45

Exponential family

An exponential family is a parametric set of probability distributions of the form��
Note that the parameter and observation variables factorize.
Examples:

Normal distribution
Bernoulli distribution
Poisson distribution
Negative binomial distribution
…

11 of 45

Link functions

There are many link functions that can be used, but there are special commonly used link functions called canonical link functions. They are special because they may have several desired properties:

Simplify derivation of the MLE.
Ensure that μ stays within the range of the outcome variable.
Provide simple sufficient statistics
…

12 of 45

Canonical link functions

13 of 45

Example: Poisson regression

Consider the case of count data:��
The dependent variable is discrete, and its variance increases with x. Furthermore, the relationship appears to be non-linear.

Example from Kida, 2019

14 of 45

Example: Poisson regression

To apply a GLM we make the following choices:

Exponential family: Poisson.
Link function: Xβ = ln(λ).�

Note that this corresponds to the following generative model for the data:

ln λ_i = β₀ + β₁x_i ,
y_i ~ Poisson(λ_i).�

Since the Poisson has only one parameter, the variance will increase linearly with the mean.
Generalized linear regression with this GLM is called Poisson regression.

15 of 45

Example: Poisson regression

Example from Kida, 2019

Prediction of the �Poisson regression

16 of 45

Logistic regression (Bernoulli regression)

To apply a GLM we make the following choices:

Exponential family: Bernoulli.
Link function: Xβ = ln(p/(1-p).�

Note that this corresponds to the following generative model for the data:

ln (p_i/ (1-p_i) = β₀ + β₁x_i ,
y_i ~ Bernoulli(p_i).�

Since the Bernoulli distribution has only one parameter p, the variance will be equal to p(1-p).

17 of 45

Inverting the link function

Note that if Xβ = ln(p/(1-p) then p/(1-p) = e^Xβ. This implies p = e^Xβ(1-p) and therefore p = e^Xβ / (1 + e^Xβ ). Equivalently, p = 1 / (1 + e^-Xβ ). This may remind you of the formula for logistic regression from Lecture 3…

steepness

midpoint

18 of 45

Recall from Lecture 3: Logistic regression for classification

The likelihood function is the product of the probability for each point:��where

19 of 45

Recall from Lecture 3: Higher dimension logistic regression

In higher dimensions the midpoint => separating hyperplane. In the example to the right, logistic regression is finding a separating line between the pink (0) and blue (1) points. The logistic function provides a measure for each point in the plane of how likely it is to be pink or blue.
In machine learning, the logistic function is called an “activation function”.
Logistic regression is a powerful and popular approach for classification.

Ntranos, Yi et al., 2019

20 of 45

Recall from Lecture 3: classification for COVID-19 tests

Booeshaghi et al., 2020

barcode + cDNA is the same structure as in single-cell RNA-seq

21 of 45

Recall from Lecture 3: classification for COVID-19 tests

Booeshaghi et al., 2020

22 of 45

Summary: regression types

Simple linear regression

Multiple linear regression

General linear models

Dependent variables

Independent variables

Generalized linear models

outcomes

outcomes:

23 of 45

General(ized) linear models

Senn: I must confess to having some confusion when I was a young statistician between general linear models and generalized linear models. Do you regret the terminology?
Nelder: I think probably I do. I suspect we should have found some more fancy name for it that would have stuck and not been confused with the general linear model, although general and generalized are not quite the same. I can see why it might have been better to have thought of something else. ��- Senn, 2003

John Nelder

24 of 45

Generalized linear models as machine learning

Generalized linear models are structured (data generating) machines:

With appropriate link functions, the exponential family condition guarantees that the likelihood function is convex. This, in turn, means that it will not display multiple local maxima, and hence is simpler to numerically optimize.
While there are constraints on the dependent variable distribution and on the link functions, the model is interpretable, i.e. the parameters have a direct and transparent connection to the observed data.�

Modern machine learning tools such as neural networks forgo even these mild parametric constraints structure in favor of direct curve fitting, albeit with more parameters. They are not directly interpretable (see Lecture 19).

25 of 45

Fitting generalized linear models in R (is easy)

Quick-R by datacamp

26 of 45

Fitting generalized linear models in R (is easy)

Quick-R by datacamp

27 of 45