Multi-task learning approaches to modeling context-specific networks

Oct 19^th, 2021

BMI 826-23 Computational Network Biology�Fall 2021

Anthony Gitter

https://compnetbiocourse.discovery.wisc.edu

Original slides created by Prof. Sushmita Roy

Strategies for capturing dynamics in networks

• Dynamic Bayesian Networks
• Skeleton network-based approaches
• Input-Output Hidden Markov Models
• Multi-task learning approaches

Goals for today

• Graphical Gaussian Models (GGMs)
• Different algorithms for learning GGMs
• Graphical Lasso
• Neighborhood selection
• Define Multi-task learning for dynamic network inference
• Learning multiple GGMs for hierarchically related tasks
• Gene Network Analysis Tool (GNAT)
• Pierson et al., 2015, PLOS Computational Biology
• Applications to inference of tissue-specific networks

Recall the univariate Gaussian distribution

• Gaussian distribution

The Gaussian distribution is defined by two parameters:

Mean:

Standard deviation:

A multi-variate Gaussian Distribution

• Extends the univariate distribution to higher dimensions (p in our case)

​

​

• As in the univariate case, we have two parameters
• Mean: a p-dimensional vector
• Co-variance: a p X p dimensional matrix
• Each entry of the matrix specifies the variance or co-variance between any two dimensions

A two-dimensional Gaussian distribution

• The mean

​

• The covariance matrix

Variance

Co-variance

Image from Mathworks.com

Probability density of a Gaussian with

A three-dimensional Gaussian distribution

https://nbviewer.org/urls/www.biostat.wisc.edu/bmi776/code/gaussian_process.ipynb

Probability density of a Gaussian with

Graphical Gaussian Models (GGMs)

• An undirected probabilistic graphical model
• Graph structure encode conditional independencies among variables
• The GGM assumes that X is drawn from a p-variate Gaussian distribution with mean and co-variance
• The graph structure specifies the zero pattern in the
• is the precision matrix
• Zero entries in the precision matrix imply absence of an edge in the graph

Absence of edges and the zero-pattern of the precision matrix

X₄

X₁

X₂

X₃

X₅

For example:

Matrix trace and determinant properties

• Trace of a pXp square matrix M is the sum of the diagonal elements

​

• Trace of two matrices

​

• For a scalar a

​

• Trace is additive

​

• Determinant of inverse

​

Joint probability of a sample from a GGM

Joint probability of a sample from a GGM

Joint probability of a sample from a GGM

Joint probability of a sample from a GGM

Joint probability of a sample from a GGM

• The previous term can be re-written as

​

Trace trick: Tr(MN)=Tr(NM)

Joint probability of a sample from a GGM

• The previous term can be re-written as

​

Trace trick: Tr(MN)=Tr(NM)

Joint probability of a sample from a GGM

• The previous term can be re-written as

​

Trace trick: Tr(MN)=Tr(NM)

Data likelihood from a GGM

• Data likelihood of a dataset D={x₁,..,x_N} with N different samples from a GGM is

​

​

• After some linear algebra is proportional to

​

• where

​

​

This formulation is nice because now we can think of entries of Θ as regression weights �that we need to maximize the above objective

Learning a Graphical Gaussian Model

• Learning the structure of a GGM entails estimating which entries in the inverse of the covariance matrix are non-zero
• These correspond to the direct dependencies among two random variables

Learning a GGM

• Graphical Lasso
• Exact approach
• Friedman, Hastie and Tibshirani 2008
• Neighborhood selection
• Approximate approach
• Meinshausen and Buhlmann 2006

Learning a GGM

• Graphical Lasso
• Exact approach
• Friedman, Hastie and Tibshirani 2008
• Neighborhood selection
• Approximate approach
• Meinshausen and Buhlmann 2006

Graphical LASSO

Graphical LASSO

• Recall the Gaussian likelihood

​

​

• Learning the GGM requires us to solve the following optimization problem

​

​

• But this in general is not going to work because of small sample size

​

​

• This is the idea behind the Graphical LASSO algorithm

​

Friedman, Hastie, Tibshirani 2008

Graphical LASSO algorithm

Keep this fixed

Graphical LASSO contd

• Using partitioned inverse

​

​

​

​

• Plugging this in

​

• For each row/column we get

Graphical LASSO contd

• This specific function looks similar to the derivative of a LASSO objective

LASSO objective

Derivative

Graphical LASSO

Learning a GGM

• Graphical Lasso
• Exact approach
• Friedman, Hastie and Tibshirani 2008
• Neighborhood selection
• Approximate approach
• Meinshausen and Buhlmann 2006

Neighborhood selection

• Proposed by Meinshausen and Buhlmann 2006
• Markov blanket: The immediate neighborhood of a random variable
• Key idea: Find the Markov blanket or immediate neighbor set of each random variable

​

X₄

X₁

X₂

X₃

X₅

Markov blanket of X₃

Neighborhood selection

• Here also we solve a set of regression problems for each random variable X_s

​

​

​

• The Markov blanket/neighborhood are those variables that have a non-zero coefficient
• Combine the neighborhood estimates using an AND or OR rule to create an undirected graph

​

​

​

​

Comparison between the two algorithms

• Neighborhood selection is fast compared to Graphical LASSO
• Neighborhood selection requires a “correction” to learn a valid structure, but this is not needed in Graphical LASSO

Goals for today

• Graphical Gaussian Models (GGMs)
• Different algorithms for learning GGMs
• Graphical Lasso
• Neighborhood selection
• Define Multi-task learning for dynamic network inference
• Learning multiple GGMs for hierarchically related tasks
• Gene Network Analysis Tool (GNAT)
• Pierson et al., 2015, PLOS Computational Biology
• Applications to inference of tissue-specific networks

Consider the following problem

• Suppose we had N time points or conditions or cell types
• We can measure p different entities for each of the cell types/individuals
• We can repeat this experiment several times (m_n)
• We wish to identify the network in each of the cell types/individuals that produces p different measurements

Genotype-Tissue Expression (GTEx) project

https://gtexportal.org/home/documentationPage

Multi-task learning (MTL)

• Suppose we had T different tasks that we want to solve
• For example, each task could be a regression task, to predict the expression level of gene from its regulator expression
• Suppose we knew the tasks are related
• Multi-task learning aims to simultaneously solve these T tasks while sharing information between them
• Different MTL frameworks might share information differently
• MTL is especially useful when for each task we do not have many samples
• GTEx network inference can be framed as MTL

​

Single task versus multi-task learning

• Single task learning

​

​

​

• Multi-task learning

Widmer and Ratsch, 2012

Loss function

Regularization term

Genetic Network Analysis Tool (GNAT)

• Given
• Gene expression measurements from multiple tissues, several per tissue
• A tissue hierarchy relating the tissues
• Do
• Learn a gene co-expression network for each tissue

​

• Naïve approach: Learn co-expression network in each tissue independently;
• Some tissues have 2 dozen samples (n<<<p)
• Key idea of GNAT is to exploit the tissue hierarchy to share information between each tissue co-expression network

Pierson et al., PLOS Computational Biology 2015

GNAT

• Each tissue’s gene network is a co-expression network: A Graphical Gaussian Model (GGM)
• Learning a GGM is equivalent to estimate the non-zeros in the inverse of the covariance matrix (precision matrix)
• Sharing information in a hierarchy by constraining the precision matrix of two tissues close on the hierarchy to be more similar to each other

​

Hierarchically related GGM learning tasks

1

m₁ samples

2

3

4

p genes

m₂ samples

m₃ samples

m₄ samples

Estimate

5

6

7

GNAT objective function

Sparse precision matrix

Encourage similarity with parent node

They don’t directly optimize this, but rather apply a two-step iterative algorithm

Parent of k from the hierarchy

K: Total number of tasks

m_k: Number of samples in task k

Two-step iterative algorithm in GNAT

• For each dataset/tissue at the leaf nodes k, learn an initial matrix Θ_k
• Repeat until convergence
• Optimize the internal matrices, Θ_p for all the ancestral nodes p keeping the leaf nodes fixed
• This can be computed analytically, because of the L2 penalty
• Optimize the leaf matrices Θ_k using their combined objective function

​

​

Updating the ancestral nodes

• To obtain the estimate of the ancestral precision matrices, we need to derive the objective with respect to each Θ_p(k)

​

​

• Turns out the ancestral matrix is an average of the child matrices

Left and right child of p

Key steps of the GNAT algorithm

Define tissue hierarchy based on gene expression levels

Learn co-expression network in each leaf tissue

Infer network in internal nodes and update leaf nodes

Final inferred networks

Tissue hierarchy used

1. Compute the mean expression of each gene per tissue
2. Tissues were clustered using hierarchical clustering of the mean expression vectors.

Results

• Ask whether sharing information helps
• Simulated data
• Apply to multi-tissue expression data from GTEX consortium
• 35 different human tissues
• 1,559 total gene expression samples
• Hierarchy learned from the expression matrix

​

​

​

Simulation experiment

• Use data from each tissue and generate five different sets
• Learn networks from four of the five datasets per tissue
• Assess the data likelihood on the hold out tests
• Baselines
• Independent learning per tissue
• Merging all datasets and learning one model
• Repeat for three gene sets

Does sharing information help?

Three gene sets. Compute test data likelihood using 5 fold cross-validation

Single network likelihood was too low to be shown!

Biological assessment of the networks

• Pairs of genes predicted to be connected were shown to be co-expressed in third party expression databases.
• Including tissue-specific expression database.
• Genes predicted to be linked in a specific tissue were 10 times more likely to be co-expressed in specific tissues
• Test if genes linked in the networks were associated with shared biological functions
• Genes that shared a function, were linked 94% more often than genes not sharing a function

​

Examining tissue-specific properties of the networks

Transcription factors specific to a tissue, tend to have a lot of connections, and connect�to genes associated with other genes specific to the tissue

Brighter the green,�the more expressed �is a gene.

Blue circles: TFs

Tissue-specific TFs (tsTFs) are highly expressed in their specific tissues

Tissues

TF groups

The signal is most apparent for Brain tissues

Additional analysis of tsTFs

• Define genes with tissue-specific functions and assess the connectivity of tsTFs to these genes versus non tissue-specific genes
• Tissue-specific genes connected to tsTFs were more expressed than genes that are not tissue-specific or genes not connected to these TFs.
• tsTFs tended to have lot of connections (hubby)
• Tissue-specific target genes were less hubby than the average gene.

Defining tissue-specific and shared gene modules

• Use a clustering algorithm to group genes into modules while using the graph structure
• We will see algorithms that do this type of clustering
• Test each module for enrichment of curated biological processes
• For each module, assess conservation in other tissues based on the fraction of links present among genes in other tissues.

Analysis of shared modules

• Define module conservation for a module m in a tissue as

​

​

Total number of tissues

Number of possible interactions among genes in the module

Number of interactions in tissue j for module m

Modules captured tissue-specific functions

An important immune-related module associated with blood-specific transcription factor GATA3. GATA3 and RUNX3 coordinately interact with other tissue-specific genes

Take away points

• Graphical Gaussian models can be used to capture direct dependencies
• Learning a GGM entails estimating the precision matrix
• Dynamics of networks: How networks change across different tissues
• GNAT: A multi-task learning approach to learn tissue-specific networks
• One task maps to a learning one GGM
• Share information between tasks using the hierarchy
• Has good generalization capability and infers biologically meaningful associations
• Gaussian assumption might be too strong

Other approaches of interest

• Ontogenet
• Jojic et al., Nature Immunology 2013
• TREEGL
• Parikh et al., Bioinformatics 2011
• Inferelator-AMuSR
• Adaptive Multiple Sparse Regression
• Castro et al., PLOS Computational Biology 2019

​

​

​

Ontogenet

• The average expression of a module is explained by a linear combination of the levels of the regulators

​

• Regulators from nearby cells on a lineage are similar

​

• Ontogenet does this by adding a penalty to the regression weights for each cell lineage.

Jojic et al., 2013

Ontogenet objective

This is the objective for a single module m, across the entire lineage

gene i in cell type t

regulator r’s activity in cell type t

{t₁,t₂) is an edge in the cell lineage tree f

TREEGL: Tree smoothed Graphical LASSO

TreeGL uses neighborhood selection to learn the graph structure

Predictive error

Sparsity penalty

Make weights similar

Inferelator-AMuSR

• S matrix: condition-specific regulator weights
• B matrix: shared regulator weights

Castro et al., PLOS Computational Biology 2019

References

• E. Pierson, the GTEx Consortium, D. Koller, A. Battle, and S. Mostafavi, “Sharing and Specificity of Co-expression Networks across 35 Human Tissues,” PLoS Comput Biol, vol. 11, no. 5, p. e1004220, May 2015, doi: 10.1371/journal.pcbi.1004220.
• V. Jojic et al., “Identification of transcriptional regulators in the mouse immune system,” Nat Immunol, vol. 14, no. 6, pp. 633–643, Jun. 2013, doi: 10.1038/ni.2587.
• A. P. Parikh, W. Wu, R. E. Curtis, and E. P. Xing, “TREEGL: reverse engineering tree-evolving gene networks underlying developing biological lineages,” Bioinformatics, vol. 27, no. 13, pp. i196–i204, Jul. 2011, doi: 10.1093/bioinformatics/btr239.
• Hastie, Trevor, Robert Tibshirani, and Martin Wainwright. 2015. Statistical Learning with Sparsity: The Lasso and Generalizations. Monographs on Statistics and Applied Probability 143. Boca Raton: CRC Press, Taylor & Francis Group. Chapter 9.

​