SPEAKER DISCLOSURE

​

​

We have not had any relationships with ACCME-defined ineligible companies within the past 24 months.

​

​

​

​

AMIA 2024 W18

Applying Generative Language Models for Biological Sequence Analysis

Presenters : Bishnu Sarker, Sayane Shome

Date: 9 November, 2024

Organizing Team and Speakers

4

Sayane Shome,Ph.D.

Postdoctoral Fellow

Anesthesia and Pediatrics,

Stanford University School of Medicine,

California, USA

sshome@stanford.edu

​

Bishnu Sarker,Ph.D.

Assistant Professor

Computer Science and Data Science,

Meharry School of Applied Computational Sciences,

Tennessee, USA

bsarker@mmc.edu

Learning Objectives

​

Educational Objectives: By the end of the tutorial, participants will:

• Learn the basics and applications of language models in bioinformatics.
• Implement Python pipeline of collecting, preprocessing, and vectorizing biological sequence data for analysis.
• Apply pre-trained transformer models and language models for specific protein sequence analysis tasks.

​

​

Link with all presentation materials/tutorials : https://sites.google.com/view/bishnusarker/invited-talks-and-tutorials/AMIA-2024-W18

5

Tutorial Agenda: Saturday, Nov 9, 2024 (1:00 – 4:30 PM)

6

Session 1

Warm-up and Refresher

Presenters : Bishnu Sarker, Sayane Shome

Date: 9 Nov, 2024

Proteins are building blocks of life

8

Amino acids and Protein structures

​

​

9

• Depending on the constituent of alkyl group in side-chain (R), there are 20 different amino acids.
• Amino acids are linked with peptide bonds leading to long polypeptide chains which fold in different manner to form tertiary structure of proteins.
• Different combinations and arrangements of amino acids result in a vast array of proteins.

Protein Domains and Protein Function

• Protein domains are distinct units within a protein that possess specific structures and functions.
• They provide modularity, flexibility, and evolutionary versatility to proteins, allowing for the development of complex functions and adaptation to diverse biological environments.
• They can be pose as sites for protein-protein interactions,metal binding sites and others.

10

Biological Sequences Are Strings.

And String Similarity is at the heart of sequence analysis.

​

11

AGGCTATCACCTGACCTCCAGGCCGATGCCC

TAGCTATCACGACCGCGGTCGATTTGCCCGAC

Given two Sequences

-AGGCTATCACCTGACCTCCAGGCCGA--TGCCC---

TAG-CTATCAC--GACCGC--GGTCGATTTGCCCGAC

Finding the sequence similarity or align them..

Introduction to Language Models and Transformers

• Language models are designed to understand and generate human language.
• Transformer-based models, such as BERT and GPT, have revolutionized this field.
• Transformers use an attention mechanism that allows them to weigh the importance of different parts of the input sequence, making them particularly powerful for tasks like text generation, translation, summarization, and more.

12

Prerequisite Python packages

1.Transformers related packages :�� For Pytorch

pip install transformers torch

For Tensorflow

pip install transformers tensorflow�

2.For handling biological sequences :

pip install biopython

​

13

Session 2

Language modelling

Presenters : Bishnu Sarker, Sayane Shome

Date: 9 Nov, 2024

What is a Language Modeling?

1. Goal: compute the probability of a sentence or sequence of words:

P(W) = P(w₁,w₂,w₃,w₄,w₅…w_n)

• Related task: probability of an upcoming word:

P(w₅|w₁,w₂,w₃,w₄)

• A model that computes either of these:

P(W) or P(w_n|w₁,w₂…w_n-1) is called a language model.

15

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

The Chain Rule applied to compute joint probability of words in a sentence.

​

​

​

​

P(“its water is so transparent”) = P(its) × P(water|its) × P(is|its water)

× P(so|its water is) × P(transparent|its water is so)

16

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Markov Assumption

​

​

In other words, we approximate each component in the product

​

​

​

​

17

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Simplest case: Unigram model

18

​

​

​

​

fifth, an, of, futures, the, an, incorporated, a, a, the, inflation, most, dollars, quarter, in, is, mass

​

thrift, did, eighty, said, hard, 'm, july, bullish

​

that, or, limited, the

Some automatically generated sentences from a Unigram model

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Bigram model

19

• Condition on the previous word:

​

​

​

​

​

texaco, rose, one, in, this, issue, is, pursuing, growth, in, a, boiler, house, said, mr., gurria, mexico, 's, motion, control, proposal, without, permission, from, five, hundred, fifty, five, yen

​

outside, new, car, parking, lot, of, the, agreement, reached

​

this, would, be, a, record, november

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

N-gram models

• We can extend to trigrams, 4-grams, 5-grams.
• In general this is an insufficient model of language because language has long-distance dependencies:

“The computer which I had just put into the machine room on the fifth floor crashed.”

• But we can often get away with N-gram models.

20

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Neural Language Models (LMs)

• Language Modeling: Calculating the probability of the next word in a sequence given some history.
• We've seen N-gram based LMs
• But neural network LMs far outperform n-gram language models
• State-of-the-art neural LMs are based on more powerful neural network technology like Transformers
• But simple feedforward LMs can do almost as well!

​

21

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Simple feedforward Neural Language Models

• Task: predict next word w_t, given prior words w_t-1, w_t-2, w_t-3, …
• Problem: Now we’re dealing with sequences of arbitrary length.
• Solution: Sliding windows (of fixed length)

22

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Neural Language Model

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Why Neural LMs work better than N-gram LMs

• Training data:
• We've seen: I have to make sure that the cat gets fed.
• Never seen: dog gets fed
• Test data:
• I forgot to make sure that the dog gets ___
• N-gram LM can't predict "fed"!
• Neural LM can use similarity of "cat" and "dog" embeddings to generalize and predict “fed” after dog

​

​

​

​

​

​

24

Jurafsky, D. and Martin, J.H. (2023) Speech and Language Processing. https://web.stanford.edu/~jurafsky/slp3/

Word2Vec Neural Language Model

25

• word2vec learns the low-rank numerical representation of words using Neural Network.
• It learns to predict the context (surrounding) words given a target word. The weights learned corresponding to each k-mer serve as the embedding.
• In natural language processing,sentences are composed of words. The spatial relations typically holds the meaning the full sentence.
• In case of biological sequence, there is no notion of words.
• K-mers such as 3-mers may serve as the words.

Session 3

Unboxing Generative Language Models

Presenters : Bishnu Sarker, Sayane Shome

Date: 9 Nov, 2024

Building Large Language Models

Raschka, S. (2024). Build a Large Language Model (From Scratch). Simon and Schuster.

”Attention is all we need”

Transformer-based Large Language Model

Transformers-based Large Language Model

29

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30.

Embedding Layer

Position Encoding

Multi-Head Attention Layer

Feed Forward Layer

Multi-Head Attention Layer

Encoder-Decoder Attention Layer

Feed Forward Neural Network

Linear Layer

Softmax Layer

Transformers Self Attention Layer

30

P

C

Alammar, J (2018). The Illustrated Transformer [Blog post]. Retrieved from https://jalammar.github.io/illustrated-transformer/

Multi-Head Attention

31

P

C

Alammar, J (2018). The Illustrated Transformer [Blog post]. Retrieved from https://jalammar.github.io/illustrated-transformer/

Attention in Sequence Analysis

Elnaggar, Ahmed, et al. "Prottrans: Toward understanding the language of life through self-supervised learning." IEEE transactions on pattern analysis and machine intelligence 44.10 (2021): 7112-7127. https://github.com/agemagician/ProtTrans

ProtTrans Architecture

33

Elnaggar, Ahmed, et al. "Prottrans: Toward understanding the language of life through self-supervised learning." IEEE transactions on pattern analysis and machine intelligence 44.10 (2021): 7112-7127.

Hands on Tutorial

Google colab notebook

34

https://tinyurl.com/yeyyfskh

For downloading all notebook related-materials :

Session 4

Hands-on case studies on applications of transformers in protein sequence analysis

Presenters : Bishnu Sarker, Sayane Shome

Date: 9 Nov, 2024

Learning Objectives

​

To expand the concepts we learnt in previous sessions into practical applications such as protein function prediction.

36

Problem Definition

​

Given a protein sequence of length L,the objective is to assign functional terms such as Gene Ontologies or Enzyme commission number.

• Gene Ontologies(GO) is a standardized system that assigns functional terms to genes and gene products based on their known or predicted molecular functions, biological processes, and cellular components.
• Enzyme Commission (EC) numbers are a classification system used to categorize enzymes based on the reactions they catalyze. The EC number provides a unique identifier for each enzyme and is widely used in biochemistry and molecular biology.

​

37

Gene Ontologies

38

Background

Manual Annotation

39

Curators

Background

Automatic Annotation

40

Protein Function Annotation

Input Data and Data Sources

41

Protein Function Annotation

Output Data and Data Sources

42

Protein Function Annotation

Approach

43

Obtaining protein sequence dataset from Uniprot and associated GO IDs/EC IDs

Evaluating ML model performance using metrics

Protein Function Annotation

Future Challenges

44

Hands on Tutorial

Google colab notebook

Link : https://tinyurl.com/yeyyfskh

1. Loading Transformer Model

1-Loading-Pre-Trainied-Transformers-From-HF.ipynb

​

• Using Embedding for Protein Function Prediction

2-Case_Study1-Transformer-Protein-Sequence-Classification.ipynb

• Fine-Tuned ESM for Protein Function Prediction

3-ESM2-Protein-Function-Prediction

45

Q&A and Final Remarks!

46

Acknowledgements

• Participants !
• AMIA 2024 Tutorial committee chairs and reviewers
• Meharry Medical College,Tennessee,USA
• Stanford University,California,USA
• National Science Foundation Grant #2302637 for supporting the travel.

​

​

47

Thank you for joining us !

For any correspondence regarding questions about the materials and related topics :

• Bishnu Sarker (bsarker@mmc.edu)
• Sayane Shome (sshome@stanford.edu)

​

​

48