Introduction to probability and statistics

Sep 16^th 2021

Slides by Prof. Sushmita Roy

Some of the material covered is adapted from BMI/CS 576 from Prof. Mark Craven

BMI 826-23 Computational Network Biology�Fall 2021

Anthony Gitter

https://compnetbiocourse.discovery.wisc.edu

Goals for today

• Introductory concepts in probability and statistics
• Introduction to probabilistic graphical models: Bayesian networks

​

RECAP from last time

• A graph has a node set and edge set
• Graphs can be undirected, weighted, directed
• Graph traversal
• Molecular networks
• Nodes are biological molecules
• Genes, proteins, metabolites, etc.
• Edges represent interaction between molecules
• Graph representations often have additional information such as sign, directionality, weight
• Example molecular networks
• Transcriptional regulatory networks, protein-protein interaction networks, signaling networks

​

Goals for today

• Introductory concepts in probability and statistics
• Introduction to probabilistic graphical models: Bayesian networks

​

​

A few key concepts

• Sample spaces
• Random variables
• Discrete and continuous distributions
• Joint, conditional, and marginal distributions
• Statistical independence
• Bayes Rule

Definition of probability

• Intuitively, we use “probability” to refer to our degree of confidence in an event of an uncertain nature.

​

• Always a number in the interval [0,1]

0 means “never occurs”

1 means “always occurs”

Sample space

• Sample space: a set of possible outcomes for an experiment
• Examples
• Flight to Chicago: {on time, late}
• Lottery: {ticket 1 wins, ticket 2 wins,…,ticket n wins}
• Weather tomorrow:

{rain, not rain} or

{sun, rain, snow} or

{sun, clouds, rain, snow, sleet}

• Roll of a die: {1,2,3,4,5,6}
• Coin toss: {Heads, Tail}

​

​

Random variables

• Random variable: maps the outcome of an uncertain experiment to a real-valued number
• A random variable can be
• Discrete/Categorical: Outcomes take a fixed set of values
• Roll of die, flight to Chicago, weather tomorrow
• Continuous: Outcomes take continuous values
• Height, weight

Notation

• Uppercase letters and words denote random variables
• X, Y
• Lowercase letters and words denote values
• x, y
• Probability that X takes value x

​

​

• We will also use the shorthand form

​

​

• For Boolean random variables, we will use the shorthand

​

​

​

​

​

​

​

​

Discrete probability distributions

Joint probability distributions

• Joint probability distribution: the function given by P(X = x, Y = y)
• Read “X equals x and Y equals y”
• Example

probability that it’s sunny

and my flight is on time

Marginal probability distributions

• The marginal distribution of X is defined by

​

​

“the distribution of X ignoring other variables”

​

• This definition generalizes to more than two variables, e.g.

Marginal distribution example

joint distribution

marginal distribution for X

Conditional distributions

• The conditional distribution of X given Y is defined as:

​

​

• Or in short

​

​

• The distribution of X given that we know the value of Y

​

• Intuitively, how much does knowing Y tell us about X?

​

​

​

Conditional distribution example

joint distribution

conditional distribution for X

given Y=on-time

Independence

• Two random variables, X and Y, are independent if

​

​

​

​

• Another way to think about this is knowing X does not tell us anything about Y 

​

Independence example #1

joint distribution

marginal distributions

Are X and Y independent here?

Independence example #1

joint distribution

marginal distributions

Are X and Y independent here?

NO.

Independence example #2

joint distribution

marginal distributions

Are X and Y independent here?

Independence example #2

joint distribution

marginal distributions

Are X and Y independent here?

YES.

Conditional independence

• Two random variables X and Y are conditionally independent given Z if 

​

​

​

“once you know the value of Z, knowing Y doesn’t tell you anything about X ”

​

• Alternatively

​

Conditional independence example

Are Fever and Headache independent?

Conditional independence example

Are Fever and Headache independent?

NO.

Conditional independence example

Are Fever and Headache conditionally independent given Flu:

Conditional independence example

Are Fever and Headache conditionally independent given Flu:

YES.

Chain rule of probability

• For two variables

​

​

• For three variables

​

​

etc.

• to see that this is true, note that

Common probability distributions

• Probability Distribution Explorer

https://distribution-explorer.github.io/

Example discrete distributions

• Binomial distribution

​

• Multinomial distribution

The binomial distribution

• Two outcomes per trial of an experiment
• Distribution over the number of successes in a fixed number n of independent trials (with same probability of success p in each)

​

​

​

• e.g. the probability of x heads in n coin flips

P(X=x)

p=0.5

p=0.1

x

x

P(X=x)

The multinomial distribution

Continuous random variables

• When our outcome is a continuous number we need a continuous random variable
• Examples: Weight, Height
• We specify a density function for random variable X as

​

​

• Probabilities are specified over an interval

​

• Probability of taking on a single value is 0

​

Continuous random variables

• To define a probability distribution for a continuous variable, we need to integrate f(x)

Example continuous distributions

• Uniform distribution

​

• Gaussian distribution

​

Uniform distribution

a

b

x

Adapted from Wikipedia

Gaussian Distribution

From Wikipedia: Normal distribution, https://en.wikipedia.org/wiki/Normal_distribution

​

Bayes rule

Posterior

Prior

Data likelihood

Marginal likelihood

• This allows us to incorporate evidence to inform our prior belief about a particular variable
• Sometimes it is easier to compute P(B|A). Bayes rule can be used to compute P(A|B).

Example of using Bayes rule

• It is harder to estimate P(Disease|Symptom) than P(Symptom|Disease)
• If we can have P(Symptom|Disease) and P(Disease), we can estimate P(Disease|Symptom)

Disease

Symptom

Goals for today

• Introductory concepts in probability and statistics
• Introduction to probabilistic graphical models: Bayesian networks

​

​

Probabilistic graphical models (PGMs)

• Combines probability and graph theory
• Nodes on the graph represent random variables
• Graph structure specifies statistical dependency structure
• Graph parameters specify the nature of the dependency
• PGMs can be directed or undirected
• Examples of PGMs: Bayesian networks, Dependency networks, Markov networks, Factor graphs

​

Different types of probabilistic graphs

• Correlation networks
• Gaussian Graphical models
• Dependency networks
• Bayesian networks

In each graph type we can assert different conditional independencies

​

Conditional independencies in PGMs

• The different classes of models we will see are based on a general notion of specifying statistical independence
• Suppose we have two genes X and Y. We add an edge between X and Y if X and Y are not independent given a third set Z.
• Depending upon Z we will have a family of different PGMs

Conditional independence and PGMs

• Correlational networks
• Z is the empty set
• Markov networks
• X and Y are not independent given all other variables
• Gaussian Graphical models are a special case (later lectures)
• Dependency networks
• Approximate Markov networks
• May not be associated with a valid joint distribution (later lectures)
• First-order conditional independence models
• Explain the correlation between two variables by a third variable
• Bayesian networks
• Generalize first-order conditional independence models

Bayesian networks (BN)

• A special type of probabilistic graphical model
• Has two parts:
• A graph which is directed and acyclic
• A set of conditional distributions
• Directed Acyclic Graph (DAG)
• The nodes denote random variables X₁… X_N
• The edges
• encode statistical dependencies between the random variables
• establish parent child relationships
• Each node X_i has a conditional probability distribution (CPD) representing P(X_i | Pa(X_i)); Pa: Parents
• Provides a tractable way to represent large joint distributions

Bayesian networks compactly represent joint distributions

An example Bayesian network

Adapted from Kevin Murphy: Intro to Graphical models and Bayes networks: http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html

Compute probabilities using a Bayesian network

What is

Compute probabilities using a Bayesian network

What is

Compute probabilities using a Bayesian network

What is

Compute probabilities using a Bayesian network

What is

Learning problems in Bayesian networks

• Given a Bayesian network B={G, Θ}
• Parameter learning
• Known graph structure G
• Given a set of joint assignments of the random variables, estimate Θ, the parameters of the CPDs
• Structure learning
• Given a set of joint assignments of the random variables, estimate the graph structure, G and parameters Θ
• Structure learning subsumes parameter learning

Estimating CPD from data

Supposed we had the following structure

C

S

R

W

And these observations for each variable

Estimating CPD from data

Supposed we had the following structure

C

S

R

W

And these observations for each variable

P(R=f|C=t)?

Parameters estimated in this way would be called the Maximum Likelihood (ML) parameters

We could put priors on the parameters and estimate a more robust set of parameters

Estimating CPD from data

Supposed we had the following structure

C

S

R

W

And these observations for each variable

? ?

? ?

? ?

? ?

P(W=t|S=t,R=f)?

Estimating CPD from data

Supposed we had the following structure

C

S

R

W

And these observations for each variable

? ?

? ?

? ?

? ?

P(W=t|S=t,R=f)?

Plan for next lectures

• Representing regulatory networks as probabilistic graphical models
• Bayesian networks
• Module networks
• Dependency networks
• Expression-based network inference
• Classes of methods
• Strengths and weaknesses of different methods

References

• Chapter 2 from Probabilistic Graphical Models. Principles and Techniques. Friedman & Koller
• Slides adapted from Prof. Mark Craven’s lectures
• Markowetz, Florian and Rainer Spang. "Inferring cellular networks-a review." BMC bioinformatics 8 Suppl 6 (2007): S5+.
• https://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html
• All of Statistics, Larry Wasserman.
• Chapter 3, The Elements of Statistical Learning, Hastie, Tibshirani, Friedman
• Probability Distribution Explorer https://distribution-explorer.github.io/