Generalised Linear Models - I

​

​

​

Saket Choudhary

saketc@iitb.ac.in

Biostatistics in Healthcare

DH 801

Lecture 12 || Tuesday, 4^th November 2025

​

Penalised Regression Cont.

dfdf

Dimensional reduction methods

• If there is a correlation structure in the predictors, we can transform the predictors and then fit a least squares model using the transformed variables
• Let Z₁, Z₂, . . . , Z_M represent M < p linear combinations of our original linear combination p predictors.
• For constants φ_1m, φ_2m . . . , φ_pm, m = 1, . . . , M, let Z_m

• Fit using OLS:

dfdf

The Goal - 1D PCA

• Reduce the “dimensionality” of data: represent the data using a small set of numbers per instance
• Perform a linear transformation of the data such that we end up retaining the maximum information
• Ideally we should be able to use the reduced dimensional space to reconstruct the original high dimensional space suffering minimum error in the process

dfdf

The Goal - 1D PCA

Source

Goal: Reduce the dimensionality of data by projecting the data into lower dimensional space such that the reconstruction error is minimized

dfdf

Minimizing reconstruction error = Maximizing variance

Original point

Projected

point

Reconstruction error

Variance

How can we determine the direction of maximal variance?

dfdf

PCs are linear combination of existing features

Source

PC1 = φ₁₁X + φ₁₂Y

PC2 = φ₂₁X + φ₂₂Y

Principal Component Analysis - Variance explained

Two equivalent interpretation:

• PCA finds the linear space with the property that the average sequared distance of the points to the linear space is minimized (“minimize reconstruction error”)�
• PCA finds the linear space such that the variance of projected points on the linear space is maximized (“maximize projected variance”)�

dfdf

PCA: the optimization

Demo

dfdf

Principal Components Regression

• Principal components regression (PCR) approach involves construct- principal components regressioning the first M principal components, Z₁, . . . , Z_M , and then using these components as the predictors in a linear regression model that is fit using least squares
• Key idea = a small number of principal components suffice to explain most of the variability in the data
• Fitting a least squares model to Z₁, . . . , Z_M than fitting a least squares model to X₁, . . . , X_p will lead to better results if the PCR assumptions are true
• Why? We can mitigate overfitting my choosing M<<p coefficients if all the information of predictors is captured in Z₁, . . . , Z_M
• Number of components determined using cross validation (to minimize test error)

dfdf

PCR: Bias variance tradeoff for determining components

​

Elastic Net

• First term encourages sparse solution to coefficients
• Second term encourages highly correlated features to be averaged

​

Linear regression

Under constraints on beta

Elastic Net

​

​

​

​

GLMs

Revisiting OLS

• In ordinary least squares (OLS or linear regression), we model the “response” (dependent) variable y as a linear combination of the “explanatory” (independent) variables x
• The fit predicts the expected value of the “response” variables as a linear combination of the “explanatory” variables
• The assumption of linear regression are usually good for variables where the variation in the mean of the response variable is small as compared to change in the explanatory variables
• However, the expected response can change by large amounts (say exponentially) with a small change in explanatory variables when the usual assumptions of OLS (which ones?) start to fail
• Example: Change in the level of snoring can increase the probability of developing a heart disease by 4x

​

​

Problem

Survey data:

​

• visits: Number of doctor consultations (outcome variable)
• gender: Male or Female
• age: Age in years (divided by 100 in raw data)
• income: Annual income in tens of thousands
• illness: Number of illnesses in past 2 weeks
• health: Self-assessed health score (0-12, higher values = better health)
• nchronic: Chronic condition indicator (no/yes)
• private: Private health insurance indicator (no/yes)

​

Problem

Problem

Problem

OLS

ls_model <- lm(visits ~ health + chronic_condition + gender + age_years + private_insurance + illness,

data = health_data)

dfdf

Linear regression: Assumptions

1. L (linear) - There is a linear relationship between the outcome variable and each covariate.
2. I (independent) - The outcome for individual observations are independent from one another, given the covariates in the model.
3. N (normal) - The residuals (errors) are normally distributed. Note that the variables themselves do not need to be normally distributed.
4. E (equal variances) - The variance of the residuals is constant across covariate groups

dfdf

What does the linear regression really model?

Independent variables (x) are measured without error, but there is error in the measurement of dependent variables (y)

​

​

​

• Probability of observing y_i is then given by:�

​

• The likelihood of observing data (x₁,...,x_n ) and (y₁,...y_n ) is given by:

​

​

​

• Conditional distribution of Y given X is Gaussian with mean: mx and variance

Generalized linear models

• Linear models assume a linear relationship between the mean “response” variable (dependent) Y and the set of “covariates” or “independent” or “explanatory” variables.
• Y is assumed to be normal
• Models linear function of the mean
• Generalized linear models: Superclass of linear models that allow modelling of non-normal response (dependent variable) as a function and allow for non-linear functions of the mean:
• Response variable: Specifies the response variable and its probability distribution
• Explanatory variables: Specifies the p explanatory variables for a linear predictor

​

​

• Link function: A function that maps the mean of the response to the explanatory variables

​

Components of a GLM

• Random Component:
• specifies the probability distribution of the response variable Y
• Example:
• Y ~ N(Xb, 𝝈²) in OLS
• Y ~ Bernoulli in the (binary) logistic regression model
• This is the only random component in the GLM → no separate error term
• Systematic Component: specifies the explanatory variables in the model, more specifically, their linear combination →
• Link function 𝞰 or g(𝞵) :
• specifies the link between the random and the systematic components
• Interpretation: How does the expected value of the response relates to the linear combination of explanatory variables
• OLS → 𝞰 = g(𝞵) = g(E[Y]) = E[Y]
• Poisson → 𝞰 = g(𝞵) = g(E[Y]) = ln(𝞵) =
• Logistic → 𝞰 = log(𝞵/(1-𝞵)) = logit(𝞵)

Advantage of GLM rather than transforming

• Before GLMs, a common way to handle response variables that violate the assumptions of OLS was to transform the variable such that the transformation was approximately normal (log/square root/…)
• However finding the transformation that achieves this is itself tricky
• It would forcefully model while we want to ideally model
• GLMs solve this problem by allowing Y to have a flexible distribution

Link function

• The probability distribution function captures how Y is distributed
• The linear predictor captures the dominance of independent factors
• The link function captures the relationship between linear predictor and the mean of the distribution function

Source

Ughhhh, this is so confusing…

• Linear model models Y=Xb+e where e is
• lm(y~x)
• GLM fits g(Y)=Xb+e where g() is the “link” function and the “e” can follow a non-normal distribution
• glm(y~x, family=’family(link=’link’))

​

g(Y)

Distribution of e

Back to the problem

Visualising GLM predictions

dfdf

Questions